Task 16143009

Name	hadcm3n_7ygn_1980_40_008455946_1
Workunit	8606802
Created	13 Dec 2013, 6:18:42 UTC
Sent	13 Dec 2013, 6:18:56 UTC
Report deadline	14 Mar 2014, 13:46:07 UTC
Received	23 Dec 2013, 22:21:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1305734
Run time	8 days 19 hours 33 min 39 sec
CPU time	5 days 15 hours 51 min 47 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7112, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:01:09 (5148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:46 (5912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:26:42 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:12 (5704): No heartbeat from core client for 30 sec - exiting 14:27:13 (5704): No heartbeat from core client for 30 sec - exiting 14:27:14 (5704): No heartbeat from core client for 30 sec - exiting 14:27:15 (5704): No heartbeat from core client for 30 sec - exiting 14:27:16 (5704): No heartbeat from core client for 30 sec - exiting 14:27:17 (5704): No heartbeat from core client for 30 sec - exiting 14:27:18 (5704): No heartbeat from core client for 30 sec - exiting 14:27:19 (5704): No heartbeat from core client for 30 sec - exiting 14:27:20 (5704): No heartbeat from core client for 30 sec - exiting 14:27:21 (5704): No heartbeat from core client for 30 sec - exiting 14:27:22 (5704): No heartbeat from core client for 30 sec - exiting 14:27:23 (5704): No heartbeat from core client for 30 sec - exiting 14:27:24 (5704): No heartbeat from core client for 30 sec - exiting 14:27:25 (5704): No heartbeat from core client for 30 sec - exiting 14:27:26 (5704): No heartbeat from core client for 30 sec - exiting 14:27:27 (5704): No heartbeat from core client for 30 sec - exiting 14:27:28 (5704): No heartbeat from core client for 30 sec - exiting 14:27:29 (5704): No heartbeat from core client for 30 sec - exiting 14:27:30 (5704): No heartbeat from core client for 30 sec - exiting 14:27:31 (5704): No heartbeat from core client for 30 sec - exiting 14:27:32 (5704): No heartbeat from core client for 30 sec - exiting 14:27:33 (5704): No heartbeat from core client for 30 sec - exiting 14:27:34 (5704): No heartbeat from core client for 30 sec - exiting 14:27:35 (5704): No heartbeat from core client for 30 sec - exiting 14:27:36 (5704): No heartbeat from core client for 30 sec - exiting 14:27:37 (5704): No heartbeat from core client for 30 sec - exiting 14:27:38 (5704): No heartbeat from core client for 30 sec - exiting 14:27:39 (5704): No heartbeat from core client for 30 sec - exiting 14:27:40 (5704): No heartbeat from core client for 30 sec - exiting 14:27:41 (5704): No heartbeat from core client for 30 sec - exiting 14:27:42 (5704): No heartbeat from core client for 30 sec - exiting 14:27:43 (5704): No heartbeat from core client for 30 sec - exiting 14:27:44 (5704): No heartbeat from core client for 30 sec - exiting 14:27:45 (5704): No heartbeat from core client for 30 sec - exiting 14:27:48 (5704): No heartbeat from core client for 30 sec - exiting 14:27:49 (5704): No heartbeat from core client for 30 sec - exiting 14:27:50 (5704): No heartbeat from core client for 30 sec - exiting 14:27:51 (5704): No heartbeat from core client for 30 sec - exiting 14:27:52 (5704): No heartbeat from core client for 30 sec - exiting 14:27:53 (5704): No heartbeat from core client for 30 sec - exiting 14:27:54 (5704): No heartbeat from core client for 30 sec - exiting 14:27:55 (5704): No heartbeat from core client for 30 sec - exiting 14:27:56 (5704): No heartbeat from core client for 30 sec - exiting 14:27:57 (5704): No heartbeat from core client for 30 sec - exiting 14:27:58 (5704): No heartbeat from core client for 30 sec - exiting 14:27:59 (5704): No heartbeat from core client for 30 sec - exiting 14:28:00 (5704): No heartbeat from core client for 30 sec - exiting 14:28:01 (5704): No heartbeat from core client for 30 sec - exiting 14:28:02 (5704): No heartbeat from core client for 30 sec - exiting 14:28:03 (5704): No heartbeat from core client for 30 sec - exiting 14:28:04 (5704): No heartbeat from core client for 30 sec - exiting 14:28:05 (5704): No heartbeat from core client for 30 sec - exiting 14:28:06 (5704): No heartbeat from core client for 30 sec - exiting 14:28:07 (5704): No heartbeat from core client for 30 sec - exiting 14:28:08 (5704): No heartbeat from core client for 30 sec - exiting 14:28:09 (5704): No heartbeat from core client for 30 sec - exiting 14:28:10 (5704): No heartbeat from core client for 30 sec - exiting 14:28:11 (5704): No heartbeat from core client for 30 sec - exiting 14:28:12 (5704): No heartbeat from core client for 30 sec - exiting 14:28:13 (5704): No heartbeat from core client for 30 sec - exiting 14:28:14 (5704): No heartbeat from core client for 30 sec - exiting 14:28:15 (5704): No heartbeat from core client for 30 sec - exiting 14:28:16 (5704): No heartbeat from core client for 30 sec - exiting 14:28:17 (5704): No heartbeat from core client for 30 sec - exiting 14:28:18 (5704): No heartbeat from core client for 30 sec - exiting 14:28:19 (5704): No heartbeat from core client for 30 sec - exiting 14:28:20 (5704): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 14:49:02 (5344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:03:40 (8072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:42 (8072): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:53:23 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5724, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 17:02:15 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:57:27 (7840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:43 (117336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:44 (117336): No heartbeat from core client for 30 sec - exiting 05:33:51 (122560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 68 to Word Address -198 Failed with Error Code -1 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Dec 2013 22:26:58	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	207,360	483,144	2.3300
20 Dec 2013 15:38:08	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	181,440	423,560	2.3344
19 Dec 2013 09:30:29	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	155,520	360,058	2.3152
18 Dec 2013 11:02:32	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	129,600	302,109	2.3311
17 Dec 2013 18:26:43	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	103,680	249,611	2.4075
16 Dec 2013 20:12:11	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	77,760	192,185	2.4715
15 Dec 2013 15:28:36	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	51,840	135,633	2.6164
14 Dec 2013 10:06:09	1305734	16143009	hadcm3n_7ygn_1980_40_008455946_1	25,920	67,348	2.5983