Task 16034564

Name	hadcm3n_80cj_1980_40_008458390_1
Workunit	8609246
Created	25 Sep 2013, 17:49:30 UTC
Sent	25 Sep 2013, 17:49:35 UTC
Report deadline	26 Dec 2013, 1:16:46 UTC
Received	3 Oct 2013, 17:31:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	731465
Run time	2 days 21 hours 48 min 10 sec
CPU time	2 days 10 hours 10 min 23 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	2.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:41:07 (1264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:47:13 (2264): No heartbeat from core client for 30 sec - exiting 14:47:14 (2264): No heartbeat from core client for 30 sec - exiting 14:47:15 (2264): No heartbeat from core client for 30 sec - exiting 14:47:16 (2264): No heartbeat from core client for 30 sec - exiting 14:47:17 (2264): No heartbeat from core client for 30 sec - exiting 14:47:18 (2264): No heartbeat from core client for 30 sec - exiting 14:47:20 (2264): No heartbeat from core client for 30 sec - exiting 14:47:21 (2264): No heartbeat from core client for 30 sec - exiting 14:47:22 (2264): No heartbeat from core client for 30 sec - exiting 14:47:23 (2264): No heartbeat from core client for 30 sec - exiting 14:47:24 (2264): No heartbeat from core client for 30 sec - exiting 14:47:25 (2264): No heartbeat from core client for 30 sec - exiting 14:47:26 (2264): No heartbeat from core client for 30 sec - exiting 14:47:27 (2264): No heartbeat from core client for 30 sec - exiting 14:47:28 (2264): No heartbeat from core client for 30 sec - exiting 14:47:30 (2264): No heartbeat from core client for 30 sec - exiting 14:47:31 (2264): No heartbeat from core client for 30 sec - exiting 14:47:32 (2264): No heartbeat from core client for 30 sec - exiting 14:47:33 (2264): No heartbeat from core client for 30 sec - exiting 14:47:34 (2264): No heartbeat from core client for 30 sec - exiting 14:47:35 (2264): No heartbeat from core client for 30 sec - exiting 14:47:36 (2264): No heartbeat from core client for 30 sec - exiting 14:47:37 (2264): No heartbeat from core client for 30 sec - exiting 14:47:39 (2264): No heartbeat from core client for 30 sec - exiting 14:47:40 (2264): No heartbeat from core client for 30 sec - exiting 14:47:41 (2264): No heartbeat from core client for 30 sec - exiting 14:47:42 (2264): No heartbeat from core client for 30 sec - exiting 14:47:43 (2264): No heartbeat from core client for 30 sec - exiting 14:47:44 (2264): No heartbeat from core client for 30 sec - exiting 14:47:45 (2264): No heartbeat from core client for 30 sec - exiting 14:47:46 (2264): No heartbeat from core client for 30 sec - exiting 14:47:48 (2264): No heartbeat from core client for 30 sec - exiting 14:47:49 (2264): No heartbeat from core client for 30 sec - exiting 14:47:50 (2264): No heartbeat from core client for 30 sec - exiting 14:47:51 (2264): No heartbeat from core client for 30 sec - exiting 14:47:52 (2264): No heartbeat from core client for 30 sec - exiting 14:47:53 (2264): No heartbeat from core client for 30 sec - exiting 14:47:54 (2264): No heartbeat from core client for 30 sec - exiting 14:47:55 (2264): No heartbeat from core client for 30 sec - exiting 14:47:57 (2264): No heartbeat from core client for 30 sec - exiting 14:47:58 (2264): No heartbeat from core client for 30 sec - exiting 14:47:59 (2264): No heartbeat from core client for 30 sec - exiting 14:48:00 (2264): No heartbeat from core client for 30 sec - exiting 14:48:01 (2264): No heartbeat from core client for 30 sec - exiting 14:48:02 (2264): No heartbeat from core client for 30 sec - exiting 14:48:03 (2264): No heartbeat from core client for 30 sec - exiting 14:48:04 (2264): No heartbeat from core client for 30 sec - exiting 14:48:06 (2264): No heartbeat from core client for 30 sec - exiting 14:48:07 (2264): No heartbeat from core client for 30 sec - exiting 14:48:08 (2264): No heartbeat from core client for 30 sec - exiting 14:48:09 (2264): No heartbeat from core client for 30 sec - exiting 14:48:10 (2264): No heartbeat from core client for 30 sec - exiting 14:48:11 (2264): No heartbeat from core client for 30 sec - exiting 14:48:13 (2264): No heartbeat from core client for 30 sec - exiting 14:48:14 (2264): No heartbeat from core client for 30 sec - exiting 14:48:16 (2264): No heartbeat from core client for 30 sec - exiting 14:48:17 (2264): No heartbeat from core client for 30 sec - exiting 14:48:18 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:19 (2264): No heartbeat from core client for 30 sec - exiting 14:48:20 (2264): No heartbeat from core client for 30 sec - exiting 14:48:21 (2264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:20:49 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:09:55 (5056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:59:53 (1560): No heartbeat from core client for 30 sec - exiting 16:59:54 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:59:56 (1560): No heartbeat from core client for 30 sec - exiting 16:59:57 (1560): No heartbeat from core client for 30 sec - exiting 16:59:58 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Oct 2013 00:27:05	731465	16034564	hadcm3n_80cj_1980_40_008458390_1	129,600	183,680	1.4173
29 Sep 2013 13:20:48	731465	16034564	hadcm3n_80cj_1980_40_008458390_1	103,680	148,198	1.4294
28 Sep 2013 15:20:31	731465	16034564	hadcm3n_80cj_1980_40_008458390_1	77,760	111,889	1.4389
27 Sep 2013 20:09:35	731465	16034564	hadcm3n_80cj_1980_40_008458390_1	51,840	73,827	1.4241
27 Sep 2013 00:04:33	731465	16034564	hadcm3n_80cj_1980_40_008458390_1	25,920	37,418	1.4436