Task 15109434

Name	hadcm3n_o0s9_2020_40_008139339_0
Workunit	8294453
Created	13 Aug 2012, 12:06:59 UTC
Sent	13 Aug 2012, 12:19:14 UTC
Report deadline	12 Nov 2012, 19:46:25 UTC
Received	11 Sep 2012, 15:12:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1230461
Run time	17 days 11 hours 24 min 24 sec
CPU time	16 days 20 hours 47 min 48 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	2.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 11:22:12 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:25:53 (8592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:25:54 (8592): No heartbeat from core client for 30 sec - exiting 11:25:55 (8592): No heartbeat from core client for 30 sec - exiting 11:25:56 (8592): No heartbeat from core client for 30 sec - exiting 11:25:57 (8592): No heartbeat from core client for 30 sec - exiting 11:25:58 (8592): No heartbeat from core client for 30 sec - exiting 11:25:59 (8592): No heartbeat from core client for 30 sec - exiting 11:26:00 (8592): No heartbeat from core client for 30 sec - exiting 11:26:01 (8592): No heartbeat from core client for 30 sec - exiting 11:26:02 (8592): No heartbeat from core client for 30 sec - exiting 11:26:03 (8592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:47:37 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o0s9ko.pjo7c10 Error converting file to netcdf: dataout/o0s9ko.pio7c10 Error converting file to netcdf: dataout/o0s9ko.pfo7c10 Error converting file to netcdf: dataout/o0s9ka.pho7c10 Error converting file to netcdf: dataout/o0s9ka.pgo7c10 Error converting file to netcdf: dataout/o0s9ka.peo7c10 Error converting file to netcdf: dataout/o0s9ka.pdo7c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:38:00 (5236): No heartbeat from core client for 30 sec - exiting 11:38:01 (5236): No heartbeat from core client for 30 sec - exiting 11:38:02 (5236): No heartbeat from core client for 30 sec - exiting 11:38:03 (5236): No heartbeat from core client for 30 sec - exiting 11:38:04 (5236): No heartbeat from core client for 30 sec - exiting 11:38:05 (5236): No heartbeat from core client for 30 sec - exiting 11:38:06 (5236): No heartbeat from core client for 30 sec - exiting 11:38:07 (5236): No heartbeat from core client for 30 sec - exiting 11:38:08 (5236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:13:31 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:00:41 (5848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5508, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5508, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Sep 2012 10:51:10	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	881,280	1,444,065	1.6386
10 Sep 2012 21:07:38	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	855,360	1,400,805	1.6377
06 Sep 2012 23:17:08	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	829,440	1,357,036	1.6361
06 Sep 2012 10:23:29	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	803,520	1,313,822	1.6351
05 Sep 2012 18:34:42	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	777,600	1,272,555	1.6365
05 Sep 2012 06:14:51	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	751,680	1,230,376	1.6368
04 Sep 2012 06:35:41	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	725,760	1,189,505	1.6390
03 Sep 2012 00:54:26	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	699,840	1,149,545	1.6426
02 Sep 2012 03:54:36	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	673,920	1,108,235	1.6445
01 Sep 2012 08:10:19	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	648,000	1,068,278	1.6486
31 Aug 2012 05:24:35	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	622,080	1,026,789	1.6506
30 Aug 2012 16:59:30	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	596,160	986,002	1.6539
30 Aug 2012 04:45:00	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	570,240	942,879	1.6535
29 Aug 2012 16:28:08	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	544,320	899,881	1.6532
29 Aug 2012 03:49:14	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	518,400	855,875	1.6510
28 Aug 2012 15:47:33	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	492,480	812,525	1.6499
28 Aug 2012 02:44:36	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	466,560	769,403	1.6491
27 Aug 2012 14:52:43	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	440,640	726,426	1.6486
27 Aug 2012 02:50:04	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	414,720	682,251	1.6451
26 Aug 2012 13:36:09	1230461	15109434	hadcm3n_o0s9_2020_40_008139339_0	388,800	638,174	1.6414