Task 15481260

Name	hadcm3n_zilr_1920_40_008256543_0
Workunit	8411667
Created	16 Dec 2012, 21:43:40 UTC
Sent	16 Dec 2012, 21:44:31 UTC
Report deadline	18 Mar 2013, 5:11:42 UTC
Received	1 Jan 2013, 21:12:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1007897
Run time	3 days 23 hours 3 min 37 sec
CPU time	3 days 17 hours 10 min 51 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7684, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:51:21 (8032): No heartbeat from core client for 30 sec - exiting 13:51:22 (8032): No heartbeat from core client for 30 sec - exiting 13:51:23 (8032): No heartbeat from core client for 30 sec - exiting 13:51:24 (8032): No heartbeat from core client for 30 sec - exiting 13:51:25 (8032): No heartbeat from core client for 30 sec - exiting 13:51:26 (8032): No heartbeat from core client for 30 sec - exiting 13:51:27 (8032): No heartbeat from core client for 30 sec - exiting 13:51:28 (8032): No heartbeat from core client for 30 sec - exiting 13:51:29 (8032): No heartbeat from core client for 30 sec - exiting 13:51:30 (8032): No heartbeat from core client for 30 sec - exiting 13:51:31 (8032): No heartbeat from core client for 30 sec - exiting 13:51:32 (8032): No heartbeat from core client for 30 sec - exiting 13:51:33 (8032): No heartbeat from core client for 30 sec - exiting 13:51:34 (8032): No heartbeat from core client for 30 sec - exiting 13:51:35 (8032): No heartbeat from core client for 30 sec - exiting 13:51:36 (8032): No heartbeat from core client for 30 sec - exiting 13:51:37 (8032): No heartbeat from core client for 30 sec - exiting 13:51:38 (8032): No heartbeat from core client for 30 sec - exiting 13:51:39 (8032): No heartbeat from core client for 30 sec - exiting 13:51:40 (8032): No heartbeat from core client for 30 sec - exiting 13:51:41 (8032): No heartbeat from core client for 30 sec - exiting 13:51:42 (8032): No heartbeat from core client for 30 sec - exiting 13:51:43 (8032): No heartbeat from core client for 30 sec - exiting 13:51:44 (8032): No heartbeat from core client for 30 sec - exiting 13:51:45 (8032): No heartbeat from core client for 30 sec - exiting 13:51:46 (8032): No heartbeat from core client for 30 sec - exiting 13:51:47 (8032): No heartbeat from core client for 30 sec - exiting 13:51:48 (8032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:09:00 (7420): No heartbeat from core client for 30 sec - exiting 13:09:01 (7420): No heartbeat from core client for 30 sec - exiting 13:09:02 (7420): No heartbeat from core client for 30 sec - exiting 13:09:03 (7420): No heartbeat from core client for 30 sec - exiting 13:09:04 (7420): No heartbeat from core client for 30 sec - exiting 13:09:05 (7420): No heartbeat from core client for 30 sec - exiting 13:09:06 (7420): No heartbeat from core client for 30 sec - exiting 13:09:07 (7420): No heartbeat from core client for 30 sec - exiting 13:09:08 (7420): No heartbeat from core client for 30 sec - exiting 13:09:09 (7420): No heartbeat from core client for 30 sec - exiting 13:09:10 (7420): No heartbeat from core client for 30 sec - exiting 13:09:11 (7420): No heartbeat from core client for 30 sec - exiting 13:09:12 (7420): No heartbeat from core client for 30 sec - exiting 13:09:13 (7420): No heartbeat from core client for 30 sec - exiting 13:09:14 (7420): No heartbeat from core client for 30 sec - exiting 13:09:15 (7420): No heartbeat from core client for 30 sec - exiting 13:09:16 (7420): No heartbeat from core client for 30 sec - exiting 13:09:17 (7420): No heartbeat from core client for 30 sec - exiting 13:09:18 (7420): No heartbeat from core client for 30 sec - exiting 13:09:19 (7420): No heartbeat from core client for 30 sec - exiting 13:09:20 (7420): No heartbeat from core client for 30 sec - exiting 13:09:21 (7420): No heartbeat from core client for 30 sec - exiting 13:09:22 (7420): No heartbeat from core client for 30 sec - exiting 13:09:23 (7420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6840, iMonCtr=1 Model crash detected, will try to restart... 13:39:15 (7440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Dec 2012 22:23:58	1007897	15481260	hadcm3n_zilr_1920_40_008256543_0	155,520	311,032	1.9999
28 Dec 2012 21:32:10	1007897	15481260	hadcm3n_zilr_1920_40_008256543_0	129,600	257,952	1.9904
26 Dec 2012 00:02:05	1007897	15481260	hadcm3n_zilr_1920_40_008256543_0	103,680	206,059	1.9875
23 Dec 2012 23:23:51	1007897	15481260	hadcm3n_zilr_1920_40_008256543_0	77,760	154,849	1.9914
21 Dec 2012 20:49:07	1007897	15481260	hadcm3n_zilr_1920_40_008256543_0	51,840	103,089	1.9886
19 Dec 2012 22:08:57	1007897	15481260	hadcm3n_zilr_1920_40_008256543_0	25,920	52,529	2.0266