Task 13028549

Name	hadcm3n_t0td_1940_40_007317287_0
Workunit	7514717
Created	29 Jun 2011, 7:17:16 UTC
Sent	29 Jun 2011, 10:08:34 UTC
Report deadline	28 Sep 2011, 17:35:45 UTC
Received	19 Jul 2011, 20:03:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1100209
Run time	6 days 18 hours 32 min 35 sec
CPU time	6 days 15 hours 46 min 25 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:31:04 (2924): No heartbeat from core client for 30 sec - exiting 15:31:05 (2924): No heartbeat from core client for 30 sec - exiting 15:31:06 (2924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:36:13 (2332): No heartbeat from core client for 30 sec - exiting 22:36:14 (2332): No heartbeat from core client for 30 sec - exiting 22:36:15 (2332): No heartbeat from core client for 30 sec - exiting 22:36:16 (2332): No heartbeat from core client for 30 sec - exiting 22:36:17 (2332): No heartbeat from core client for 30 sec - exiting 22:36:18 (2332): No heartbeat from core client for 30 sec - exiting 22:36:20 (2332): No heartbeat from core client for 30 sec - exiting 22:36:21 (2332): No heartbeat from core client for 30 sec - exiting 22:36:22 (2332): No heartbeat from core client for 30 sec - exiting 22:36:23 (2332): No heartbeat from core client for 30 sec - exiting 22:36:24 (2332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:50 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:45:21 (3872): No heartbeat from core client for 30 sec - exiting 22:45:22 (3872): No heartbeat from core client for 30 sec - exiting 22:45:23 (3872): No heartbeat from core client for 30 sec - exiting 22:45:24 (3872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:12:11 (2716): No heartbeat from core client for 30 sec - exiting 14:12:12 (2716): No heartbeat from core client for 30 sec - exiting 14:12:13 (2716): No heartbeat from core client for 30 sec - exiting 14:12:14 (2716): No heartbeat from core client for 30 sec - exiting 14:12:15 (2716): No heartbeat from core client for 30 sec - exiting 14:12:16 (2716): No heartbeat from core client for 30 sec - exiting 14:12:17 (2716): No heartbeat from core client for 30 sec - exiting 14:12:18 (2716): No heartbeat from core client for 30 sec - exiting 14:12:19 (2716): No heartbeat from core client for 30 sec - exiting 14:12:20 (2716): No heartbeat from core client for 30 sec - exiting 14:12:21 (2716): No heartbeat from core client for 30 sec - exiting 14:12:22 (2716): No heartbeat from core client for 30 sec - exiting 14:12:23 (2716): No heartbeat from core client for 30 sec - exiting 14:12:24 (2716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:44:16 (312): No heartbeat from core client for 30 sec - exiting 05:44:18 (312): No heartbeat from core client for 30 sec - exiting 05:44:19 (312): No heartbeat from core client for 30 sec - exiting 05:44:20 (312): No heartbeat from core client for 30 sec - exiting 05:44:21 (312): No heartbeat from core client for 30 sec - exiting 05:44:22 (312): No heartbeat from core client for 30 sec - exiting 05:44:23 (312): No heartbeat from core client for 30 sec - exiting 05:44:24 (312): No heartbeat from core client for 30 sec - exiting 05:44:25 (312): No heartbeat from core client for 30 sec - exiting 05:44:26 (312): No heartbeat from core client for 30 sec - exiting 05:44:27 (312): No heartbeat from core client for 30 sec - exiting 05:44:28 (312): No heartbeat from core client for 30 sec - exiting 05:44:30 (312): No heartbeat from core client for 30 sec - exiting 05:44:31 (312): No heartbeat from core client for 30 sec - exiting 05:44:32 (312): No heartbeat from core client for 30 sec - exiting 05:44:33 (312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:44:34 (312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 05:52:42 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5120, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:44:31 (1020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:46:46 (3032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:48:33 (1020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:50:59 (2896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:29 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:24 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:43:25 (200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:46:13 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:46:14 (4052): No heartbeat from core client for 30 sec - exiting 15:47:17 (1364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:54 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:04 (2844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:50 (2756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:42:21 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:56:48 (2008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:46 (3420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:45:02 (2844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:10 (3008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 14:50:58 (2288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 17:57:45	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	259,200	537,084	2.0721
25 Jul 2011 17:34:44	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	233,280	484,345	2.0762
25 Jul 2011 16:43:24	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	207,360	432,010	2.0834
25 Jul 2011 16:10:46	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	181,440	376,267	2.0738
25 Jul 2011 15:45:00	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	155,520	319,503	2.0544
25 Jul 2011 14:57:05	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	129,600	264,152	2.0382
03 Jul 2011 00:25:53	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	103,680	214,239	2.0663
02 Jul 2011 23:24:55	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	77,760	164,464	2.1150
01 Jul 2011 23:09:29	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	51,840	111,689	2.1545
30 Jun 2011 21:03:21	1100209	13028549	hadcm3n_t0td_1940_40_007317287_0	25,920	56,365	2.1746