Task 15863227

Name	hadcm3n_3elp_1980_40_008395161_0
Workunit	8546020
Created	25 Jun 2013, 13:00:29 UTC
Sent	30 Jun 2013, 16:13:49 UTC
Report deadline	29 Sep 2013, 23:41:00 UTC
Received	14 Aug 2013, 19:14:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1285741
Run time	7 days 9 hours 6 min 38 sec
CPU time	2 days 4 hours 36 min 14 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:50:07 (18032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:29:42 (21572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:08:30 (21848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:08:31 (21848): No heartbeat from core client for 30 sec - exiting 05:08:32 (21848): No heartbeat from core client for 30 sec - exiting 05:08:33 (21848): No heartbeat from core client for 30 sec - exiting 05:08:34 (21848): No heartbeat from core client for 30 sec - exiting 05:59:22 (31540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:37 (15996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:38 (15996): No heartbeat from core client for 30 sec - exiting 06:01:39 (15996): No heartbeat from core client for 30 sec - exiting 06:01:40 (15996): No heartbeat from core client for 30 sec - exiting 06:01:41 (15996): No heartbeat from core client for 30 sec - exiting 06:01:42 (15996): No heartbeat from core client for 30 sec - exiting 06:01:43 (15996): No heartbeat from core client for 30 sec - exiting 06:01:44 (15996): No heartbeat from core client for 30 sec - exiting 06:01:45 (15996): No heartbeat from core client for 30 sec - exiting 06:01:46 (15996): No heartbeat from core client for 30 sec - exiting 06:01:47 (15996): No heartbeat from core client for 30 sec - exiting 06:18:33 (13636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:35 (13636): No heartbeat from core client for 30 sec - exiting 11:34:10 (27588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:12 (27588): No heartbeat from core client for 30 sec - exiting 14:15:27 (15104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:28 (15104): No heartbeat from core client for 30 sec - exiting 16:19:29 (10448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:19:30 (10448): No heartbeat from core client for 30 sec - exiting 19:28:58 (12672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:59 (12672): No heartbeat from core client for 30 sec - exiting 20:34:51 (24724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:52 (24724): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 23:12:43 (2968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:12:44 (2968): No heartbeat from core client for 30 sec - exiting 23:12:45 (2968): No heartbeat from core client for 30 sec - exiting 16:05:23 (3368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:06:17 (7016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:21 (7016): No heartbeat from core client for 30 sec - exiting 01:45:21 (1816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:31 (1816): No heartbeat from core client for 30 sec - exiting 02:01:40 (1292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:42 (1292): No heartbeat from core client for 30 sec - exiting 02:01:43 (1292): No heartbeat from core client for 30 sec - exiting 02:56:04 (10320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:08 (10320): No heartbeat from core client for 30 sec - exiting 02:56:09 (10320): No heartbeat from core client for 30 sec - exiting 02:56:10 (10320): No heartbeat from core client for 30 sec - exiting 02:56:11 (10320): No heartbeat from core client for 30 sec - exiting 03:02:45 (8204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:46 (8204): No heartbeat from core client for 30 sec - exiting 03:16:49 (11064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:51 (11064): No heartbeat from core client for 30 sec - exiting 03:16:52 (11064): No heartbeat from core client for 30 sec - exiting 03:32:09 (9044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:07 (6880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:10 (6880): No heartbeat from core client for 30 sec - exiting 03:55:13 (6880): No heartbeat from core client for 30 sec - exiting 03:55:14 (6880): No heartbeat from core client for 30 sec - exiting 04:01:55 (9040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:17:17 (7336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:17:22 (7336): No heartbeat from core client for 30 sec - exiting 04:17:23 (7336): No heartbeat from core client for 30 sec - exiting 04:17:24 (7336): No heartbeat from core client for 30 sec - exiting 04:17:25 (7336): No heartbeat from core client for 30 sec - exiting 04:32:01 (8928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:03 (8928): No heartbeat from core client for 30 sec - exiting 05:45:57 (6356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:00 (6356): No heartbeat from core client for 30 sec - exiting 06:31:56 (7812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:58 (7812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 12:19:10 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:16 (6784): No heartbeat from core client for 30 sec - exiting 12:19:17 (6784): No heartbeat from core client for 30 sec - exiting 12:51:05 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:51:06 (5468): No heartbeat from core client for 30 sec - exiting 12:51:07 (5468): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5656, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6704, iMonCtr=1 Model crash detected, will try to restart... 23:18:08 (3356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:11 (3356): No heartbeat from core client for 30 sec - exiting 23:34:42 (9268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:34:43 (9268): No heartbeat from core client for 30 sec - exiting 23:34:44 (9268): No heartbeat from core client for 30 sec - exiting 00:43:44 (8872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:43:45 (8872): No heartbeat from core client for 30 sec - exiting 01:18:15 (8820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:17 (8820): No heartbeat from core client for 30 sec - exiting 01:18:18 (8820): No heartbeat from core client for 30 sec - exiting 01:35:31 (3396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:34 (3396): No heartbeat from core client for 30 sec - exiting 01:35:35 (3396): No heartbeat from core client for 30 sec - exiting 01:36:48 (888): No heartbeat from core client for 30 sec - exiting 01:36:49 (888): No heartbeat from core client for 30 sec - exiting 01:36:50 (888): No heartbeat from core client for 30 sec - exiting 01:36:51 (888): No heartbeat from core client for 30 sec - exiting 01:36:52 (888): No heartbeat from core client for 30 sec - exiting 01:36:53 (888): No heartbeat from core client for 30 sec - exiting 01:36:54 (888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:58 (8320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:00 (8320): No heartbeat from core client for 30 sec - exiting 02:18:01 (8320): No heartbeat from core client for 30 sec - exiting 02:33:52 (3108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:54 (3108): No heartbeat from core client for 30 sec - exiting 04:47:57 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:18:03 (12568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:18:05 (12568): No heartbeat from core client for 30 sec - exiting 08:18:02 (17188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:04 (17188): No heartbeat from core client for 30 sec - exiting 16:13:04 (5724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:32 (39864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:34 (39864): No heartbeat from core client for 30 sec - exiting 21:57:35 (39864): No heartbeat from core client for 30 sec - exiting 21:57:36 (39864): No heartbeat from core client for 30 sec - exiting 21:57:37 (39864): No heartbeat from core client for 30 sec - exiting 22:18:02 (39200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:18:04 (39200): No heartbeat from core client for 30 sec - exiting 22:18:05 (39200): No heartbeat from core client for 30 sec - exiting 22:21:21 (40008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:10 (39444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:12 (39444): No heartbeat from core client for 30 sec - exiting 23:59:13 (39444): No heartbeat from core client for 30 sec - exiting 23:59:14 (39444): No heartbeat from core client for 30 sec - exiting 23:59:15 (39444): No heartbeat from core client for 30 sec - exiting 18:35:37 (4168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:49:39 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:05 (6956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:15 (6956): No heartbeat from core client for 30 sec - exiting 17:30:16 (6956): No heartbeat from core client for 30 sec - exiting 17:30:17 (6956): No heartbeat from core client for 30 sec - exiting 17:30:18 (6956): No heartbeat from core client for 30 sec - exiting 17:30:19 (6956): No heartbeat from core client for 30 sec - exiting 17:30:20 (6956): No heartbeat from core client for 30 sec - exiting 17:44:38 (6664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:43 (6664): No heartbeat from core client for 30 sec - exiting 17:44:44 (6664): No heartbeat from core client for 30 sec - exiting 17:44:45 (6664): No heartbeat from core client for 30 sec - exiting 17:44:46 (6664): No heartbeat from core client for 30 sec - exiting 17:44:47 (6664): No heartbeat from core client for 30 sec - exiting 17:44:48 (6664): No heartbeat from core client for 30 sec - exiting 17:44:49 (6664): No heartbeat from core client for 30 sec - exiting 17:44:50 (6664): No heartbeat from core client for 30 sec - exiting 17:44:51 (6664): No heartbeat from core client for 30 sec - exiting 17:44:52 (6664): No heartbeat from core client for 30 sec - exiting 17:44:53 (6664): No heartbeat from core client for 30 sec - exiting 17:44:54 (6664): No heartbeat from core client for 30 sec - exiting 17:44:55 (6664): No heartbeat from core client for 30 sec - exiting 17:44:56 (6664): No heartbeat from core client for 30 sec - exiting 17:44:57 (6664): No heartbeat from core client for 30 sec - exiting 17:44:58 (6664): No heartbeat from core client for 30 sec - exiting 17:44:59 (6664): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 18:44:11 (2988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:13 (2988): No heartbeat from core client for 30 sec - exiting 18:44:14 (2988): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 20:46:39 (6596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:46:40 (6596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 04:27:23 (17968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:27:25 (17968): No heartbeat from core client for 30 sec - exiting 04:27:26 (17968): No heartbeat from core client for 30 sec - exiting 04:27:27 (17968): No heartbeat from core client for 30 sec - exiting 04:27:28 (17968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:30:47 (39528): No heartbeat from core client for 30 sec - exiting 14:30:48 (39528): No heartbeat from core client for 30 sec - exiting 14:30:49 (39528): No heartbeat from core client for 30 sec - exiting 14:30:50 (39528): No heartbeat from core client for 30 sec - exiting 14:30:51 (39528): No heartbeat from core client for 30 sec - exiting 14:30:52 (39528): No heartbeat from core client for 30 sec - exiting 14:30:53 (39528): No heartbeat from core client for 30 sec - exiting 14:30:54 (39528): No heartbeat from core client for 30 sec - exiting 14:30:55 (39528): No heartbeat from core client for 30 sec - exiting 14:30:56 (39528): No heartbeat from core client for 30 sec - exiting 14:30:58 (39528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:30:44 (16316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:52:41 (41824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:13:47 (51868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:13:48 (51868): No heartbeat from core client for 30 sec - exiting 17:30:49 (50000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:52 (54656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:56 (54656): No heartbeat from core client for 30 sec - exiting 18:30:57 (54656): No heartbeat from core client for 30 sec - exiting 18:30:59 (54656): No heartbeat from core client for 30 sec - exiting 18:31:00 (54656): No heartbeat from core client for 30 sec - exiting 18:31:01 (54656): No heartbeat from core client for 30 sec - exiting 18:31:02 (54656): No heartbeat from core client for 30 sec - exiting 18:31:03 (54656): No heartbeat from core client for 30 sec - exiting 19:26:22 (5248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:24 (5248): No heartbeat from core client for 30 sec - exiting 20:20:10 (50488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:12 (50488): No heartbeat from core client for 30 sec - exiting 20:30:53 (46784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:53 (56332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:54 (56332): No heartbeat from core client for 30 sec - exiting 22:39:18 (58480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:39:22 (58480): No heartbeat from core client for 30 sec - exiting 22:39:23 (58480): No heartbeat from core client for 30 sec - exiting 23:30:54 (15032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:56 (15032): No heartbeat from core client for 30 sec - exiting 23:30:57 (15032): No heartbeat from core client for 30 sec - exiting 23:30:58 (15032): No heartbeat from core client for 30 sec - exiting 23:30:59 (15032): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 23:37:38 (24160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:39 (24160): No heartbeat from core client for 30 sec - exiting 23:37:40 (24160): No heartbeat from core client for 30 sec - exiting 23:37:41 (24160): No heartbeat from core client for 30 sec - exiting 23:37:42 (24160): No heartbeat from core client for 30 sec - exiting 23:37:43 (24160): No heartbeat from core client for 30 sec - exiting 23:37:44 (24160): No heartbeat from core client for 30 sec - exiting 23:37:45 (24160): No heartbeat from core client for 30 sec - exiting 23:37:46 (24160): No heartbeat from core client for 30 sec - exiting 23:37:47 (24160): No heartbeat from core client for 30 sec - exiting 23:37:48 (24160): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 00:00:48 (34048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:50 (34048): No heartbeat from core client for 30 sec - exiting 00:30:51 (47560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:52 (13964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:53 (13964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 19:19:02	1285741	15863227	hadcm3n_3elp_1980_40_008395161_0	77,760	185,687	2.3880
23 Jul 2013 14:56:31	1285741	15863227	hadcm3n_3elp_1980_40_008395161_0	51,840	126,697	2.4440
23 Jul 2013 14:56:31	1285741	15863227	hadcm3n_3elp_1980_40_008395161_0	25,920	62,865	2.4253