Task 13087059

Name	hadcm3n_g039_1940_40_007297957_2
Workunit	7495381
Created	5 Jul 2011, 10:20:44 UTC
Sent	5 Jul 2011, 10:20:48 UTC
Report deadline	4 Oct 2011, 17:47:59 UTC
Received	16 Jul 2011, 6:16:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	7 days 23 hours 56 min 22 sec
CPU time	7 days 6 hours 29 min 14 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	1.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:00:17 (6440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:06:12 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 17:12:23 (9252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:24 (9252): No heartbeat from core client for 30 sec - exiting 17:12:25 (9252): No heartbeat from core client for 30 sec - exiting 17:12:26 (9252): No heartbeat from core client for 30 sec - exiting 17:12:27 (9252): No heartbeat from core client for 30 sec - exiting 17:14:40 (5584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:15:37 (9200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 17:23:51 (7256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:52 (7256): No heartbeat from core client for 30 sec - exiting 17:23:53 (7256): No heartbeat from core client for 30 sec - exiting 17:23:54 (7256): No heartbeat from core client for 30 sec - exiting 17:23:55 (7256): No heartbeat from core client for 30 sec - exiting 17:23:56 (7256): No heartbeat from core client for 30 sec - exiting 17:23:58 (7256): No heartbeat from core client for 30 sec - exiting 17:23:59 (7256): No heartbeat from core client for 30 sec - exiting 17:24:00 (7256): No heartbeat from core client for 30 sec - exiting 17:24:01 (7256): No heartbeat from core client for 30 sec - exiting 17:24:02 (7256): No heartbeat from core client for 30 sec - exiting 17:26:41 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 17:33:58 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 22:59:34 (3408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:59:35 (3408): No heartbeat from core client for 30 sec - exiting 22:59:36 (3408): No heartbeat from core client for 30 sec - exiting 22:59:37 (3408): No heartbeat from core client for 30 sec - exiting 22:59:38 (3408): No heartbeat from core client for 30 sec - exiting 22:59:39 (3408): No heartbeat from core client for 30 sec - exiting 22:59:40 (3408): No heartbeat from core client for 30 sec - exiting 22:59:42 (3408): No heartbeat from core client for 30 sec - exiting 22:59:43 (3408): No heartbeat from core client for 30 sec - exiting 22:59:44 (3408): No heartbeat from core client for 30 sec - exiting 22:59:45 (3408): No heartbeat from core client for 30 sec - exiting 23:13:04 (7052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:20:10 (4000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:20:50 (4000): No heartbeat from core client for 30 sec - exiting 23:20:51 (4000): No heartbeat from core client for 30 sec - exiting 23:20:52 (4000): No heartbeat from core client for 30 sec - exiting 23:20:53 (4000): No heartbeat from core client for 30 sec - exiting 23:20:54 (4000): No heartbeat from core client for 30 sec - exiting 23:20:55 (4000): No heartbeat from core client for 30 sec - exiting 23:20:56 (4000): No heartbeat from core client for 30 sec - exiting 23:20:57 (4000): No heartbeat from core client for 30 sec - exiting 23:20:58 (4000): No heartbeat from core client for 30 sec - exiting 23:21:00 (4000): No heartbeat from core client for 30 sec - exiting 23:26:06 (2836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:33:09 (6628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:33:10 (6628): No heartbeat from core client for 30 sec - exiting 23:33:11 (6628): No heartbeat from core client for 30 sec - exiting 23:33:12 (6628): No heartbeat from core client for 30 sec - exiting 23:33:14 (6628): No heartbeat from core client for 30 sec - exiting 23:33:15 (6628): No heartbeat from core client for 30 sec - exiting 23:33:16 (6628): No heartbeat from core client for 30 sec - exiting 23:33:17 (6628): No heartbeat from core client for 30 sec - exiting 23:33:18 (6628): No heartbeat from core client for 30 sec - exiting 23:33:19 (6628): No heartbeat from core client for 30 sec - exiting 23:33:20 (6628): No heartbeat from core client for 30 sec - exiting 23:36:20 (7060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:36:37 (7060): No heartbeat from core client for 30 sec - exiting 23:36:38 (7060): No heartbeat from core client for 30 sec - exiting 23:36:39 (7060): No heartbeat from core client for 30 sec - exiting 23:36:40 (7060): No heartbeat from core client for 30 sec - exiting 23:36:41 (7060): No heartbeat from core client for 30 sec - exiting 23:36:42 (7060): No heartbeat from core client for 30 sec - exiting 23:36:43 (7060): No heartbeat from core client for 30 sec - exiting 23:36:44 (7060): No heartbeat from core client for 30 sec - exiting 23:39:11 (6676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:32 (6940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:03 (1840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:04 (1840): No heartbeat from core client for 30 sec - exiting 23:42:05 (1840): No heartbeat from core client for 30 sec - exiting 23:42:06 (1840): No heartbeat from core client for 30 sec - exiting 23:42:07 (1840): No heartbeat from core client for 30 sec - exiting 23:42:09 (1840): No heartbeat from core client for 30 sec - exiting 23:42:10 (1840): No heartbeat from core client for 30 sec - exiting 23:42:11 (1840): No heartbeat from core client for 30 sec - exiting 23:42:12 (1840): No heartbeat from core client for 30 sec - exiting 23:42:13 (1840): No heartbeat from core client for 30 sec - exiting 23:42:14 (1840): No heartbeat from core client for 30 sec - exiting 23:43:54 (332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:45:25 (316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:45:29 (316): No heartbeat from core client for 30 sec - exiting 23:45:30 (316): No heartbeat from core client for 30 sec - exiting 23:45:31 (316): No heartbeat from core client for 30 sec - exiting 23:45:32 (316): No heartbeat from core client for 30 sec - exiting 23:45:33 (316): No heartbeat from core client for 30 sec - exiting 23:45:34 (316): No heartbeat from core client for 30 sec - exiting 23:45:35 (316): No heartbeat from core client for 30 sec - exiting 23:45:36 (316): No heartbeat from core client for 30 sec - exiting 23:45:37 (316): No heartbeat from core client for 30 sec - exiting 23:45:39 (316): No heartbeat from core client for 30 sec - exiting 23:48:58 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:57:01 (2872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:08 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:21 (6180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:22 (6180): No heartbeat from core client for 30 sec - exiting 00:04:23 (6180): No heartbeat from core client for 30 sec - exiting 00:04:24 (6180): No heartbeat from core client for 30 sec - exiting 00:04:25 (6180): No heartbeat from core client for 30 sec - exiting 00:04:26 (6180): No heartbeat from core client for 30 sec - exiting 00:04:27 (6180): No heartbeat from core client for 30 sec - exiting 00:04:28 (6180): No heartbeat from core client for 30 sec - exiting 00:04:29 (6180): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 00:29:47 (7004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 14:59:37	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	233,280	675,815	2.8970
25 Jul 2011 13:15:50	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	207,360	602,522	2.9057
25 Jul 2011 13:15:50	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	181,440	538,564	2.9683
25 Jul 2011 13:15:50	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	155,520	468,132	3.0101
25 Jul 2011 13:15:50	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	129,600	396,653	3.0606
25 Jul 2011 13:15:50	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	103,680	317,196	3.0594
10 Jul 2011 19:02:41	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	77,760	237,627	3.0559
09 Jul 2011 20:36:55	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	51,840	158,330	3.0542
08 Jul 2011 22:15:23	1122757	13087059	hadcm3n_g039_1940_40_007297957_2	25,920	78,962	3.0464