Task 15439767

Name	hadcm3n_o6wr_2100_40_008245286_0
Workunit	8400410
Created	20 Nov 2012, 17:49:51 UTC
Sent	20 Nov 2012, 17:50:00 UTC
Report deadline	20 Feb 2013, 1:17:11 UTC
Received	19 Dec 2012, 2:17:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1155778
Run time	3 days 6 hours 15 min 58 sec
CPU time	3 days 3 hours 31 min 31 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	2.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:07:38 (10272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:19:07 (4708): No heartbeat from core client for 30 sec - exiting 14:19:08 (4708): No heartbeat from core client for 30 sec - exiting 14:19:09 (4708): No heartbeat from core client for 30 sec - exiting 14:19:10 (4708): No heartbeat from core client for 30 sec - exiting 14:19:11 (4708): No heartbeat from core client for 30 sec - exiting 14:19:12 (4708): No heartbeat from core client for 30 sec - exiting 14:19:13 (4708): No heartbeat from core client for 30 sec - exiting 14:19:14 (4708): No heartbeat from core client for 30 sec - exiting 14:19:15 (4708): No heartbeat from core client for 30 sec - exiting 14:19:16 (4708): No heartbeat from core client for 30 sec - exiting 14:19:17 (4708): No heartbeat from core client for 30 sec - exiting 14:19:18 (4708): No heartbeat from core client for 30 sec - exiting 14:19:19 (4708): No heartbeat from core client for 30 sec - exiting 14:19:20 (4708): No heartbeat from core client for 30 sec - exiting 14:19:21 (4708): No heartbeat from core client for 30 sec - exiting 14:19:22 (4708): No heartbeat from core client for 30 sec - exiting 14:19:24 (4708): No heartbeat from core client for 30 sec - exiting 14:19:25 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=968, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=968, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=968, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=968, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:23:09 (5440): No heartbeat from core client for 30 sec - exiting 12:23:10 (5440): No heartbeat from core client for 30 sec - exiting 12:23:11 (5440): No heartbeat from core client for 30 sec - exiting 12:23:12 (5440): No heartbeat from core client for 30 sec - exiting 12:23:13 (5440): No heartbeat from core client for 30 sec - exiting 12:23:14 (5440): No heartbeat from core client for 30 sec - exiting 12:23:15 (5440): No heartbeat from core client for 30 sec - exiting 12:23:16 (5440): No heartbeat from core client for 30 sec - exiting 12:23:17 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:16 (8984): No heartbeat from core client for 30 sec - exiting 15:46:17 (8984): No heartbeat from core client for 30 sec - exiting 15:46:18 (8984): No heartbeat from core client for 30 sec - exiting 15:46:19 (8984): No heartbeat from core client for 30 sec - exiting 15:46:20 (8984): No heartbeat from core client for 30 sec - exiting 15:46:21 (8984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:24:25 (11608): No heartbeat from core client for 30 sec - exiting 07:24:27 (11608): No heartbeat from core client for 30 sec - exiting 07:24:28 (11608): No heartbeat from core client for 30 sec - exiting 07:24:29 (11608): No heartbeat from core client for 30 sec - exiting 07:24:30 (11608): No heartbeat from core client for 30 sec - exiting 07:24:31 (11608): No heartbeat from core client for 30 sec - exiting 07:24:32 (11608): No heartbeat from core client for 30 sec - exiting 07:24:33 (11608): No heartbeat from core client for 30 sec - exiting 07:24:34 (11608): No heartbeat from core client for 30 sec - exiting 07:24:35 (11608): No heartbeat from core client for 30 sec - exiting 07:24:36 (11608): No heartbeat from core client for 30 sec - exiting 07:24:37 (11608): No heartbeat from core client for 30 sec - exiting 07:24:38 (11608): No heartbeat from core client for 30 sec - exiting 07:24:39 (11608): No heartbeat from core client for 30 sec - exiting 07:24:40 (11608): No heartbeat from core client for 30 sec - exiting 07:24:41 (11608): No heartbeat from core client for 30 sec - exiting 07:24:42 (11608): No heartbeat from core client for 30 sec - exiting 07:24:43 (11608): No heartbeat from core client for 30 sec - exiting 07:24:44 (11608): No heartbeat from core client for 30 sec - exiting 07:24:45 (11608): No heartbeat from core client for 30 sec - exiting 07:24:46 (11608): No heartbeat from core client for 30 sec - exiting 07:24:47 (11608): No heartbeat from core client for 30 sec - exiting 07:24:48 (11608): No heartbeat from core client for 30 sec - exiting 07:24:49 (11608): No heartbeat from core client for 30 sec - exiting 07:24:50 (11608): No heartbeat from core client for 30 sec - exiting 07:24:51 (11608): No heartbeat from core client for 30 sec - exiting 07:24:52 (11608): No heartbeat from core client for 30 sec - exiting 07:24:53 (11608): No heartbeat from core client for 30 sec - exiting 07:24:54 (11608): No heartbeat from core client for 30 sec - exiting 07:24:55 (11608): No heartbeat from core client for 30 sec - exiting 07:24:56 (11608): No heartbeat from core client for 30 sec - exiting 07:24:57 (11608): No heartbeat from core client for 30 sec - exiting 07:24:58 (11608): No heartbeat from core client for 30 sec - exiting 07:24:59 (11608): No heartbeat from core client for 30 sec - exiting 07:25:00 (11608): No heartbeat from core client for 30 sec - exiting 07:25:01 (11608): No heartbeat from core client for 30 sec - exiting 07:25:02 (11608): No heartbeat from core client for 30 sec - exiting 07:25:03 (11608): No heartbeat from core client for 30 sec - exiting 07:25:04 (11608): No heartbeat from core client for 30 sec - exiting 07:25:05 (11608): No heartbeat from core client for 30 sec - exiting 07:25:06 (11608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12996, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12996, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Dec 2012 20:25:42	1155778	15439767	hadcm3n_o6wr_2100_40_008245286_0	129,600	239,265	1.8462
16 Dec 2012 20:25:42	1155778	15439767	hadcm3n_o6wr_2100_40_008245286_0	103,680	189,259	1.8254
02 Dec 2012 07:53:27	1155778	15439767	hadcm3n_o6wr_2100_40_008245286_0	77,760	139,113	1.7890
21 Nov 2012 19:42:37	1155778	15439767	hadcm3n_o6wr_2100_40_008245286_0	51,840	91,558	1.7662
21 Nov 2012 06:43:19	1155778	15439767	hadcm3n_o6wr_2100_40_008245286_0	25,920	45,470	1.7542