Task 13557645

Name	hadcm3n_yadk_1900_40_007525945_1
Workunit	7723420
Created	28 Oct 2011, 13:43:27 UTC
Sent	29 Oct 2011, 21:47:52 UTC
Report deadline	29 Jan 2012, 5:15:03 UTC
Received	14 Dec 2011, 19:45:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1177240
Run time	14 days 23 hours 2 min 40 sec
CPU time	14 days 12 hours 29 min 56 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.56 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=105256, iMonCtr=1 Model crash detected, will try to restart... 03:01:35 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:36 (4032): No heartbeat from core client for 30 sec - exiting 03:01:37 (4032): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3728, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2836, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5076, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=264, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:57:27 (9128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:29 (9128): No heartbeat from core client for 30 sec - exiting 00:57:31 (9128): No heartbeat from core client for 30 sec - exiting 00:57:32 (9128): No heartbeat from core client for 30 sec - exiting 00:59:13 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:38 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:39 (3312): No heartbeat from core client for 30 sec - exiting 01:21:04 (5588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:21:06 (5588): No heartbeat from core client for 30 sec - exiting 01:21:07 (5588): No heartbeat from core client for 30 sec - exiting 01:22:44 (7676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:12 (7668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7492, selfPID=7492, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yadkko.pjc1c10 Error converting file to netcdf: dataout/yadkko.pic1c10 Error converting file to netcdf: dataout/yadkko.pfc1c10 Error converting file to netcdf: dataout/yadkka.phc1c10 Error converting file to netcdf: dataout/yadkka.pgc1c10 Error converting file to netcdf: dataout/yadkka.pec1c10 Error converting file to netcdf: dataout/yadkka.pdc1c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yadkko.pjc1c10 Error converting file to netcdf: dataout/yadkko.pic1c10 Error converting file to netcdf: dataout/yadkko.pfc1c10 Error converting file to netcdf: dataout/yadkka.phc1c10 Error converting file to netcdf: dataout/yadkka.pgc1c10 Error converting file to netcdf: dataout/yadkka.pec1c10 Error converting file to netcdf: dataout/yadkka.pdc1c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3004, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3004, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4124, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Dec 2011 19:47:49	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	777,600	1,254,594	1.6134
13 Dec 2011 01:28:25	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	751,680	1,209,589	1.6092
11 Dec 2011 14:53:20	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	725,760	1,164,581	1.6046
11 Dec 2011 02:00:11	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	699,840	1,119,142	1.5991
10 Dec 2011 01:34:19	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	673,920	1,073,010	1.5922
07 Dec 2011 20:18:28	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	648,000	1,025,706	1.5829
04 Dec 2011 20:02:04	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	622,080	978,337	1.5727
03 Dec 2011 22:02:03	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	596,160	933,753	1.5663
02 Dec 2011 19:46:52	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	570,240	888,533	1.5582
27 Nov 2011 22:22:13	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	518,400	797,080	1.5376
27 Nov 2011 00:08:07	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	492,480	752,266	1.5275
25 Nov 2011 22:16:39	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	466,560	705,480	1.5121
21 Nov 2011 20:15:04	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	440,640	658,845	1.4952
18 Nov 2011 18:21:58	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	414,720	612,567	1.4771
16 Nov 2011 03:17:24	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	388,800	566,582	1.4573
15 Nov 2011 17:31:23	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	362,880	523,620	1.4430
15 Nov 2011 17:31:23	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	336,960	491,813	1.4596
15 Nov 2011 17:31:23	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	311,040	451,976	1.4531
15 Nov 2011 17:31:23	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	285,120	413,128	1.4490
10 Nov 2011 01:15:44	1177240	13557645	hadcm3n_yadk_1900_40_007525945_1	259,200	368,492	1.4217