Task 16078363

Name	hadcm3n_79n3_1980_40_008423778_3
Workunit	8574634
Created	6 Nov 2013, 21:48:31 UTC
Sent	6 Nov 2013, 21:48:41 UTC
Report deadline	6 Feb 2014, 5:15:52 UTC
Received	8 Nov 2013, 5:40:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	731465
Run time	19 hours 1 min 3 sec
CPU time	15 hours 11 min 45 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.61 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/79n3ko.pji1c10 Error converting file to netcdf: dataout/79n3ko.pii1c10 Error converting file to netcdf: dataout/79n3ko.pfi1c10 Error converting file to netcdf: dataout/79n3ka.phi1c10 Error converting file to netcdf: dataout/79n3ka.pgi1c10 Error converting file to netcdf: dataout/79n3ka.pei1c10 Error converting file to netcdf: dataout/79n3ka.pdi1c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:33:53 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:35:57 (4976): No heartbeat from core client for 30 sec - exiting 11:36:12 (4976): No heartbeat from core client for 30 sec - exiting 11:38:13 (6620): No heartbeat from core client for 30 sec - exiting 11:38:14 (6620): No heartbeat from core client for 30 sec - exiting 11:38:15 (6620): No heartbeat from core client for 30 sec - exiting 11:38:16 (6620): No heartbeat from core client for 30 sec - exiting 11:38:17 (6620): No heartbeat from core client for 30 sec - exiting 11:38:19 (6620): No heartbeat from core client for 30 sec - exiting 11:38:20 (6620): No heartbeat from core client for 30 sec - exiting 11:38:21 (6620): No heartbeat from core client for 30 sec - exiting 11:38:22 (6620): No heartbeat from core client for 30 sec - exiting 11:38:23 (6620): No heartbeat from core client for 30 sec - exiting 11:38:24 (6620): No heartbeat from core client for 30 sec - exiting 11:38:25 (6620): No heartbeat from core client for 30 sec - exiting 11:38:26 (6620): No heartbeat from core client for 30 sec - exiting 11:38:28 (6620): No heartbeat from core client for 30 sec - exiting 11:38:29 (6620): No heartbeat from core client for 30 sec - exiting 11:38:30 (6620): No heartbeat from core client for 30 sec - exiting 11:38:31 (6620): No heartbeat from core client for 30 sec - exiting 11:38:32 (6620): No heartbeat from core client for 30 sec - exiting 11:38:33 (6620): No heartbeat from core client for 30 sec - exiting 11:38:34 (6620): No heartbeat from core client for 30 sec - exiting 11:38:35 (6620): No heartbeat from core client for 30 sec - exiting 11:38:37 (6620): No heartbeat from core client for 30 sec - exiting 11:38:38 (6620): No heartbeat from core client for 30 sec - exiting 11:38:39 (6620): No heartbeat from core client for 30 sec - exiting 11:38:40 (6620): No heartbeat from core client for 30 sec - exiting 11:38:41 (6620): No heartbeat from core client for 30 sec - exiting 11:38:42 (6620): No heartbeat from core client for 30 sec - exiting 11:38:43 (6620): No heartbeat from core client for 30 sec - exiting 11:38:44 (6620): No heartbeat from core client for 30 sec - exiting 11:38:45 (6620): No heartbeat from core client for 30 sec - exiting 11:38:47 (6620): No heartbeat from core client for 30 sec - exiting 11:38:48 (6620): No heartbeat from core client for 30 sec - exiting 11:38:49 (6620): No heartbeat from core client for 30 sec - exiting 11:38:50 (6620): No heartbeat from core client for 30 sec - exiting 11:38:51 (6620): No heartbeat from core client for 30 sec - exiting 11:38:52 (6620): No heartbeat from core client for 30 sec - exiting 11:38:53 (6620): No heartbeat from core client for 30 sec - exiting 11:38:55 (6620): No heartbeat from core client for 30 sec - exiting 11:38:56 (6620): No heartbeat from core client for 30 sec - exiting 11:38:57 (6620): No heartbeat from core client for 30 sec - exiting 11:38:58 (6620): No heartbeat from core client for 30 sec - exiting 11:38:59 (6620): No heartbeat from core client for 30 sec - exiting 11:39:00 (6620): No heartbeat from core client for 30 sec - exiting 11:39:01 (6620): No heartbeat from core client for 30 sec - exiting 11:39:02 (6620): No heartbeat from core client for 30 sec - exiting 11:39:03 (6620): No heartbeat from core client for 30 sec - exiting 11:39:05 (6620): No heartbeat from core client for 30 sec - exiting 11:39:06 (6620): No heartbeat from core client for 30 sec - exiting 11:39:07 (6620): No heartbeat from core client for 30 sec - exiting 11:39:08 (6620): No heartbeat from core client for 30 sec - exiting 11:39:09 (6620): No heartbeat from core client for 30 sec - exiting 11:39:10 (6620): No heartbeat from core client for 30 sec - exiting 11:39:11 (6620): No heartbeat from core client for 30 sec - exiting 11:39:12 (6620): No heartbeat from core client for 30 sec - exiting 11:39:14 (6620): No heartbeat from core client for 30 sec - exiting 11:39:15 (6620): No heartbeat from core client for 30 sec - exiting 11:39:16 (6620): No heartbeat from core client for 30 sec - exiting 11:39:17 (6620): No heartbeat from core client for 30 sec - exiting 11:39:18 (6620): No heartbeat from core client for 30 sec - exiting 11:39:19 (6620): No heartbeat from core client for 30 sec - exiting 11:39:20 (6620): No heartbeat from core client for 30 sec - exiting 11:39:21 (6620): No heartbeat from core client for 30 sec - exiting 11:39:23 (6620): No heartbeat from core client for 30 sec - exiting 11:39:24 (6620): No heartbeat from core client for 30 sec - exiting 11:39:25 (6620): No heartbeat from core client for 30 sec - exiting 11:39:26 (6620): No heartbeat from core client for 30 sec - exiting 11:39:27 (6620): No heartbeat from core client for 30 sec - exiting 11:39:28 (6620): No heartbeat from core client for 30 sec - exiting 11:39:29 (6620): No heartbeat from core client for 30 sec - exiting 11:39:30 (6620): No heartbeat from core client for 30 sec - exiting 11:39:32 (6620): No heartbeat from core client for 30 sec - exiting 11:39:33 (6620): No heartbeat from core client for 30 sec - exiting 11:39:34 (6620): No heartbeat from core client for 30 sec - exiting 11:39:35 (6620): No heartbeat from core client for 30 sec - exiting 11:39:36 (6620): No heartbeat from core client for 30 sec - exiting 11:39:37 (6620): No heartbeat from core client for 30 sec - exiting 11:39:38 (6620): No heartbeat from core client for 30 sec - exiting 11:39:39 (6620): No heartbeat from core client for 30 sec - exiting 11:39:40 (6620): No heartbeat from core client for 30 sec - exiting 11:39:42 (6620): No heartbeat from core client for 30 sec - exiting 11:39:43 (6620): No heartbeat from core client for 30 sec - exiting 11:39:44 (6620): No heartbeat from core client for 30 sec - exiting 11:39:45 (6620): No heartbeat from core client for 30 sec - exiting 11:39:47 (6620): No heartbeat from core client for 30 sec - exiting 11:39:48 (6620): No heartbeat from core client for 30 sec - exiting 11:39:49 (6620): No heartbeat from core client for 30 sec - exiting 11:39:50 (6620): No heartbeat from core client for 30 sec - exiting 11:39:51 (6620): No heartbeat from core client for 30 sec - exiting 11:39:52 (6620): No heartbeat from core client for 30 sec - exiting 11:39:53 (6620): No heartbeat from core client for 30 sec - exiting 11:39:55 (6620): No heartbeat from core client for 30 sec - exiting 11:39:56 (6620): No heartbeat from core client for 30 sec - exiting 11:39:57 (6620): No heartbeat from core client for 30 sec - exiting 11:39:58 (6620): No heartbeat from core client for 30 sec - exiting 11:39:59 (6620): No heartbeat from core client for 30 sec - exiting 11:40:00 (6620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:40:01 (6620): No heartbeat from core client for 30 sec - exiting 11:40:02 (6620): No heartbeat from core client for 30 sec - exiting 11:40:04 (6620): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Nov 2013 13:51:50	731465	16078363	hadcm3n_79n3_1980_40_008423778_3	25,920	37,868	1.4610