Task 12987419

Name	hadcm3n_s0y7_1940_40_007298477_2
Workunit	7495901
Created	20 Jun 2011, 14:11:02 UTC
Sent	20 Jun 2011, 14:11:06 UTC
Report deadline	19 Sep 2011, 21:38:17 UTC
Received	4 Jul 2011, 9:39:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	431496
Run time	9 days 22 hours 33 min 46 sec
CPU time	7 days 13 hours 56 min 33 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	1.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:49:36 (6244): No heartbeat from core client for 30 sec - exiting 16:49:37 (6244): No heartbeat from core client for 30 sec - exiting 16:49:39 (6244): No heartbeat from core client for 30 sec - exiting 16:49:40 (6244): No heartbeat from core client for 30 sec - exiting 16:49:41 (6244): No heartbeat from core client for 30 sec - exiting 16:49:42 (6244): No heartbeat from core client for 30 sec - exiting 16:49:43 (6244): No heartbeat from core client for 30 sec - exiting 16:49:44 (6244): No heartbeat from core client for 30 sec - exiting 16:49:45 (6244): No heartbeat from core client for 30 sec - exiting 16:49:46 (6244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:16:28 (1088): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:07:23 (7416): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:39:25 (6944): No heartbeat from core client for 30 sec - exiting 18:39:26 (6944): No heartbeat from core client for 30 sec - exiting 18:39:27 (6944): No heartbeat from core client for 30 sec - exiting 18:39:28 (6944): No heartbeat from core client for 30 sec - exiting 18:39:29 (6944): No heartbeat from core client for 30 sec - exiting 18:39:30 (6944): No heartbeat from core client for 30 sec - exiting 18:39:31 (6944): No heartbeat from core client for 30 sec - exiting 18:39:32 (6944): No heartbeat from core client for 30 sec - exiting 18:39:34 (6944): No heartbeat from core client for 30 sec - exiting 18:39:35 (6944): No heartbeat from core client for 30 sec - exiting 18:39:36 (6944): No heartbeat from core client for 30 sec - exiting 18:39:37 (6944): No heartbeat from core client for 30 sec - exiting 18:39:38 (6944): No heartbeat from core client for 30 sec - exiting 18:39:39 (6944): No heartbeat from core client for 30 sec - exiting 18:39:40 (6944): No heartbeat from core client for 30 sec - exiting 18:39:41 (6944): No heartbeat from core client for 30 sec - exiting 18:39:42 (6944): No heartbeat from core client for 30 sec - exiting 18:39:43 (6944): No heartbeat from core client for 30 sec - exiting 18:39:44 (6944): No heartbeat from core client for 30 sec - exiting 18:39:46 (6944): No heartbeat from core client for 30 sec - exiting 18:39:47 (6944): No heartbeat from core client for 30 sec - exiting 18:39:48 (6944): No heartbeat from core client for 30 sec - exiting 18:39:49 (6944): No heartbeat from core client for 30 sec - exiting 18:39:50 (6944): No heartbeat from core client for 30 sec - exiting 18:39:51 (6944): No heartbeat from core client for 30 sec - exiting 18:39:52 (6944): No heartbeat from core client for 30 sec - exiting 18:39:53 (6944): No heartbeat from core client for 30 sec - exiting 18:39:54 (6944): No heartbeat from core client for 30 sec - exiting 18:39:55 (6944): No heartbeat from core client for 30 sec - exiting 18:39:57 (6944): No heartbeat from core client for 30 sec - exiting 18:39:58 (6944): No heartbeat from core client for 30 sec - exiting 18:39:59 (6944): No heartbeat from core client for 30 sec - exiting 18:40:00 (6944): No heartbeat from core client for 30 sec - exiting 18:40:01 (6944): No heartbeat from core client for 30 sec - exiting 18:40:02 (6944): No heartbeat from core client for 30 sec - exiting 18:40:03 (6944): No heartbeat from core client for 30 sec - exiting 18:40:04 (6944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:52:14 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:38:36 (8832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:58 (6856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jul 2011 06:41:54	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	233,280	802,908	3.4418
03 Jul 2011 04:59:19	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	207,360	714,400	3.4452
02 Jul 2011 01:21:16	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	181,440	625,298	3.4463
30 Jun 2011 21:08:24	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	155,520	535,953	3.4462
29 Jun 2011 18:58:39	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	129,600	446,278	3.4435
28 Jun 2011 06:52:59	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	103,680	356,461	3.4381
27 Jun 2011 05:12:52	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	77,760	268,631	3.4546
22 Jun 2011 21:02:40	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	51,840	179,698	3.4664
21 Jun 2011 17:30:35	431496	12987419	hadcm3n_s0y7_1940_40_007298477_2	25,920	89,703	3.4608