Task 16006102

Name	hadcm3n_o210_1980_40_008387803_4
Workunit	8538662
Created	6 Sep 2013, 21:42:02 UTC
Sent	6 Sep 2013, 22:06:55 UTC
Report deadline	7 Dec 2013, 5:34:06 UTC
Received	10 Sep 2013, 6:03:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291985
Run time	2 days 20 hours 27 min 5 sec
CPU time	2 days 19 hours 4 min 16 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:36:53 (4228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:43:27 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:29 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 13:28:51 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:34 (1084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:45:39 (2352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:45:40 (2352): No heartbeat from core client for 30 sec - exiting 14:45:41 (2352): No heartbeat from core client for 30 sec - exiting 14:45:42 (2352): No heartbeat from core client for 30 sec - exiting 14:45:43 (2352): No heartbeat from core client for 30 sec - exiting 14:45:44 (2352): No heartbeat from core client for 30 sec - exiting 14:45:45 (2352): No heartbeat from core client for 30 sec - exiting 14:45:46 (2352): No heartbeat from core client for 30 sec - exiting 14:45:47 (2352): No heartbeat from core client for 30 sec - exiting 14:45:48 (2352): No heartbeat from core client for 30 sec - exiting 14:45:49 (2352): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 15:04:24 (5864): No heartbeat from core client for 30 sec - exiting 15:04:25 (5864): No heartbeat from core client for 30 sec - exiting 15:04:26 (5864): No heartbeat from core client for 30 sec - exiting 15:04:27 (5864): No heartbeat from core client for 30 sec - exiting 15:04:28 (5864): No heartbeat from core client for 30 sec - exiting 15:04:29 (5864): No heartbeat from core client for 30 sec - exiting 15:04:30 (5864): No heartbeat from core client for 30 sec - exiting 15:04:31 (5864): No heartbeat from core client for 30 sec - exiting 15:04:32 (5864): No heartbeat from core client for 30 sec - exiting 15:04:33 (5864): No heartbeat from core client for 30 sec - exiting 15:04:34 (5864): No heartbeat from core client for 30 sec - exiting 15:04:35 (5864): No heartbeat from core client for 30 sec - exiting 15:04:36 (5864): No heartbeat from core client for 30 sec - exiting 15:04:37 (5864): No heartbeat from core client for 30 sec - exiting 15:04:38 (5864): No heartbeat from core client for 30 sec - exiting 15:04:39 (5864): No heartbeat from core client for 30 sec - exiting 15:04:40 (5864): No heartbeat from core client for 30 sec - exiting 15:04:41 (5864): No heartbeat from core client for 30 sec - exiting 15:04:42 (5864): No heartbeat from core client for 30 sec - exiting 15:04:43 (5864): No heartbeat from core client for 30 sec - exiting 15:04:44 (5864): No heartbeat from core client for 30 sec - exiting 15:04:45 (5864): No heartbeat from core client for 30 sec - exiting 15:04:46 (5864): No heartbeat from core client for 30 sec - exiting 15:04:47 (5864): No heartbeat from core client for 30 sec - exiting 15:04:48 (5864): No heartbeat from core client for 30 sec - exiting 15:04:49 (5864): No heartbeat from core client for 30 sec - exiting 15:04:50 (5864): No heartbeat from core client for 30 sec - exiting 15:04:51 (5864): No heartbeat from core client for 30 sec - exiting 15:04:52 (5864): No heartbeat from core client for 30 sec - exiting 15:04:53 (5864): No heartbeat from core client for 30 sec - exiting 15:04:54 (5864): No heartbeat from core client for 30 sec - exiting 15:04:55 (5864): No heartbeat from core client for 30 sec - exiting 15:04:56 (5864): No heartbeat from core client for 30 sec - exiting 15:04:57 (5864): No heartbeat from core client for 30 sec - exiting 15:04:58 (5864): No heartbeat from core client for 30 sec - exiting 15:04:59 (5864): No heartbeat from core client for 30 sec - exiting 15:05:00 (5864): No heartbeat from core client for 30 sec - exiting 15:05:01 (5864): No heartbeat from core client for 30 sec - exiting 15:05:02 (5864): No heartbeat from core client for 30 sec - exiting 15:05:03 (5864): No heartbeat from core client for 30 sec - exiting 15:05:04 (5864): No heartbeat from core client for 30 sec - exiting 15:05:05 (5864): No heartbeat from core client for 30 sec - exiting 15:05:06 (5864): No heartbeat from core client for 30 sec - exiting 15:05:07 (5864): No heartbeat from core client for 30 sec - exiting 15:05:08 (5864): No heartbeat from core client for 30 sec - exiting 15:05:09 (5864): No heartbeat from core client for 30 sec - exiting 15:05:10 (5864): No heartbeat from core client for 30 sec - exiting 15:05:11 (5864): No heartbeat from core client for 30 sec - exiting 15:05:12 (5864): No heartbeat from core client for 30 sec - exiting 15:05:13 (5864): No heartbeat from core client for 30 sec - exiting 15:05:14 (5864): No heartbeat from core client for 30 sec - exiting 15:05:15 (5864): No heartbeat from core client for 30 sec - exiting 15:05:16 (5864): No heartbeat from core client for 30 sec - exiting 15:05:17 (5864): No heartbeat from core client for 30 sec - exiting 15:05:18 (5864): No heartbeat from core client for 30 sec - exiting 15:05:19 (5864): No heartbeat from core client for 30 sec - exiting 15:05:20 (5864): No heartbeat from core client for 30 sec - exiting 15:05:21 (5864): No heartbeat from core client for 30 sec - exiting 15:05:22 (5864): No heartbeat from core client for 30 sec - exiting 15:05:23 (5864): No heartbeat from core client for 30 sec - exiting 15:05:24 (5864): No heartbeat from core client for 30 sec - exiting 15:05:25 (5864): No heartbeat from core client for 30 sec - exiting 15:05:26 (5864): No heartbeat from core client for 30 sec - exiting 15:05:27 (5864): No heartbeat from core client for 30 sec - exiting 15:05:28 (5864): No heartbeat from core client for 30 sec - exiting 15:05:29 (5864): No heartbeat from core client for 30 sec - exiting 15:05:30 (5864): No heartbeat from core client for 30 sec - exiting 15:05:31 (5864): No heartbeat from core client for 30 sec - exiting 15:05:32 (5864): No heartbeat from core client for 30 sec - exiting 15:05:33 (5864): No heartbeat from core client for 30 sec - exiting 15:05:34 (5864): No heartbeat from core client for 30 sec - exiting 15:05:35 (5864): No heartbeat from core client for 30 sec - exiting 15:05:36 (5864): No heartbeat from core client for 30 sec - exiting 15:05:37 (5864): No heartbeat from core client for 30 sec - exiting 15:05:38 (5864): No heartbeat from core client for 30 sec - exiting 15:05:39 (5864): No heartbeat from core client for 30 sec - exiting 15:05:40 (5864): No heartbeat from core client for 30 sec - exiting 15:05:41 (5864): No heartbeat from core client for 30 sec - exiting 15:05:42 (5864): No heartbeat from core client for 30 sec - exiting 15:05:43 (5864): No heartbeat from core client for 30 sec - exiting 15:05:44 (5864): No heartbeat from core client for 30 sec - exiting 15:05:45 (5864): No heartbeat from core client for 30 sec - exiting 15:05:46 (5864): No heartbeat from core client for 30 sec - exiting 15:05:47 (5864): No heartbeat from core client for 30 sec - exiting 15:05:48 (5864): No heartbeat from core client for 30 sec - exiting 15:05:49 (5864): No heartbeat from core client for 30 sec - exiting 15:05:50 (5864): No heartbeat from core client for 30 sec - exiting 15:05:51 (5864): No heartbeat from core client for 30 sec - exiting 15:05:52 (5864): No heartbeat from core client for 30 sec - exiting 15:05:53 (5864): No heartbeat from core client for 30 sec - exiting 15:05:54 (5864): No heartbeat from core client for 30 sec - exiting 15:05:55 (5864): No heartbeat from core client for 30 sec - exiting 15:05:56 (5864): No heartbeat from core client for 30 sec - exiting 15:05:57 (5864): No heartbeat from core client for 30 sec - exiting 15:05:58 (5864): No heartbeat from core client for 30 sec - exiting 15:05:59 (5864): No heartbeat from core client for 30 sec - exiting 15:06:00 (5864): No heartbeat from core client for 30 sec - exiting 15:06:01 (5864): No heartbeat from core client for 30 sec - exiting 15:06:02 (5864): No heartbeat from core client for 30 sec - exiting 15:06:03 (5864): No heartbeat from core client for 30 sec - exiting 15:06:04 (5864): No heartbeat from core client for 30 sec - exiting 15:06:05 (5864): No heartbeat from core client for 30 sec - exiting 15:06:06 (5864): No heartbeat from core client for 30 sec - exiting 15:06:07 (5864): No heartbeat from core client for 30 sec - exiting 15:06:08 (5864): No heartbeat from core client for 30 sec - exiting 15:06:09 (5864): No heartbeat from core client for 30 sec - exiting 15:06:10 (5864): No heartbeat from core client for 30 sec - exiting 15:06:11 (5864): No heartbeat from core client for 30 sec - exiting 15:06:12 (5864): No heartbeat from core client for 30 sec - exiting 15:06:13 (5864): No heartbeat from core client for 30 sec - exiting 15:06:14 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:07:42 (5536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Sep 2013 14:26:18	1291985	16006102	hadcm3n_o210_1980_40_008387803_4	103,680	225,315	2.1732
08 Sep 2013 22:40:49	1291985	16006102	hadcm3n_o210_1980_40_008387803_4	77,760	168,867	2.1716
08 Sep 2013 06:41:23	1291985	16006102	hadcm3n_o210_1980_40_008387803_4	51,840	113,964	2.1984
07 Sep 2013 14:30:28	1291985	16006102	hadcm3n_o210_1980_40_008387803_4	25,920	58,213	2.2459