Task 12989265

Name	hadcm3n_s2tq_1940_40_007299211_2
Workunit	7496635
Created	20 Jun 2011, 18:28:02 UTC
Sent	20 Jun 2011, 18:28:03 UTC
Report deadline	20 Sep 2011, 1:55:14 UTC
Received	25 Jun 2011, 11:16:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1106244
Run time	4 days 14 hours 21 min 52 sec
CPU time	4 days 8 hours 43 min 52 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	3.02 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 21:04:20 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:30 (4484): No heartbeat from core client for 30 sec - exiting 21:05:31 (4484): No heartbeat from core client for 30 sec - exiting 21:05:32 (4484): No heartbeat from core client for 30 sec - exiting 21:05:33 (4484): No heartbeat from core client for 30 sec - exiting 21:05:34 (4484): No heartbeat from core client for 30 sec - exiting 21:05:35 (4484): No heartbeat from core client for 30 sec - exiting 21:05:36 (4484): No heartbeat from core client for 30 sec - exiting 21:05:37 (4484): No heartbeat from core client for 30 sec - exiting 21:05:38 (4484): No heartbeat from core client for 30 sec - exiting 21:05:39 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:13:55 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:05:41 (992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:06:40 (992): No heartbeat from core client for 30 sec - exiting 22:03:35 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:42 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:15 (3752): No heartbeat from core client for 30 sec - exiting 05:41:32 (3752): No heartbeat from core client for 30 sec - exiting 05:41:39 (3752): No heartbeat from core client for 30 sec - exiting 05:41:40 (3752): No heartbeat from core client for 30 sec - exiting 05:41:41 (3752): No heartbeat from core client for 30 sec - exiting 05:41:42 (3752): No heartbeat from core client for 30 sec - exiting 05:41:43 (3752): No heartbeat from core client for 30 sec - exiting 05:41:44 (3752): No heartbeat from core client for 30 sec - exiting 05:41:45 (3752): No heartbeat from core client for 30 sec - exiting 05:41:46 (3752): No heartbeat from core client for 30 sec - exiting 05:41:47 (3752): No heartbeat from core client for 30 sec - exiting 05:41:48 (3752): No heartbeat from core client for 30 sec - exiting 05:41:49 (3752): No heartbeat from core client for 30 sec - exiting 05:41:50 (3752): No heartbeat from core client for 30 sec - exiting 05:41:51 (3752): No heartbeat from core client for 30 sec - exiting 05:42:09 (3752): No heartbeat from core client for 30 sec - exiting 05:42:16 (3752): No heartbeat from core client for 30 sec - exiting 05:44:10 (3752): No heartbeat from core client for 30 sec - exiting 13:02:47 (4580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:05:42 (4580): No heartbeat from core client for 30 sec - exiting 13:05:43 (4580): No heartbeat from core client for 30 sec - exiting 13:05:44 (4580): No heartbeat from core client for 30 sec - exiting 13:05:45 (4580): No heartbeat from core client for 30 sec - exiting 13:05:46 (4580): No heartbeat from core client for 30 sec - exiting 13:05:47 (4580): No heartbeat from core client for 30 sec - exiting 13:05:48 (4580): No heartbeat from core client for 30 sec - exiting 13:05:49 (4580): No heartbeat from core client for 30 sec - exiting 13:05:50 (4580): No heartbeat from core client for 30 sec - exiting 13:05:51 (4580): No heartbeat from core client for 30 sec - exiting 13:05:52 (4580): No heartbeat from core client for 30 sec - exiting 13:05:53 (4580): No heartbeat from core client for 30 sec - exiting 13:05:54 (4580): No heartbeat from core client for 30 sec - exiting 13:05:55 (4580): No heartbeat from core client for 30 sec - exiting 13:05:56 (4580): No heartbeat from core client for 30 sec - exiting 13:05:57 (4580): No heartbeat from core client for 30 sec - exiting 13:05:58 (4580): No heartbeat from core client for 30 sec - exiting 13:05:59 (4580): No heartbeat from core client for 30 sec - exiting 13:06:00 (4580): No heartbeat from core client for 30 sec - exiting 13:06:01 (4580): No heartbeat from core client for 30 sec - exiting 13:06:02 (4580): No heartbeat from core client for 30 sec - exiting 13:06:03 (4580): No heartbeat from core client for 30 sec - exiting 13:06:04 (4580): No heartbeat from core client for 30 sec - exiting 13:06:05 (4580): No heartbeat from core client for 30 sec - exiting 13:06:06 (4580): No heartbeat from core client for 30 sec - exiting 13:06:07 (4580): No heartbeat from core client for 30 sec - exiting 13:06:08 (4580): No heartbeat from core client for 30 sec - exiting 13:06:09 (4580): No heartbeat from core client for 30 sec - exiting 13:06:10 (4580): No heartbeat from core client for 30 sec - exiting 13:06:11 (4580): No heartbeat from core client for 30 sec - exiting 13:06:12 (4580): No heartbeat from core client for 30 sec - exiting 13:06:13 (4580): No heartbeat from core client for 30 sec - exiting 13:06:14 (4580): No heartbeat from core client for 30 sec - exiting 13:06:15 (4580): No heartbeat from core client for 30 sec - exiting 13:06:16 (4580): No heartbeat from core client for 30 sec - exiting 13:06:17 (4580): No heartbeat from core client for 30 sec - exiting 13:06:18 (4580): No heartbeat from core client for 30 sec - exiting 13:56:52 (3036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:17:42 (4076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:22 (4076): No heartbeat from core client for 30 sec - exiting 04:21:39 (4076): No heartbeat from core client for 30 sec - exiting 04:21:40 (4076): No heartbeat from core client for 30 sec - exiting 04:21:41 (4076): No heartbeat from core client for 30 sec - exiting 04:21:42 (4076): No heartbeat from core client for 30 sec - exiting 04:21:43 (4076): No heartbeat from core client for 30 sec - exiting 04:21:44 (4076): No heartbeat from core client for 30 sec - exiting 12:03:55 (4640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:04:14 (4640): No heartbeat from core client for 30 sec - exiting 20:07:03 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:04 (4800): No heartbeat from core client for 30 sec - exiting 20:10:07 (4800): No heartbeat from core client for 30 sec - exiting 20:10:09 (4800): No heartbeat from core client for 30 sec - exiting 20:57:30 (2300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:48:46 (1800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 12:17:33 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jun 2011 10:18:33	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	233,280	376,377	1.6134
25 Jun 2011 01:09:19	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	207,360	342,608	1.6522
24 Jun 2011 11:38:08	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	181,440	299,449	1.6504
23 Jun 2011 22:40:00	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	155,520	255,800	1.6448
23 Jun 2011 09:39:22	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	129,600	212,313	1.6382
22 Jun 2011 20:52:35	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	103,680	169,013	1.6301
22 Jun 2011 08:01:56	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	77,760	125,683	1.6163
21 Jun 2011 19:50:22	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	51,840	82,479	1.5910
21 Jun 2011 07:08:56	1106244	12989265	hadcm3n_s2tq_1940_40_007299211_2	25,920	39,108	1.5088