Task 13103281

Name	hadcm3n_yd01_1900_40_007349707_0
Workunit	7547137
Created	6 Jul 2011, 14:00:34 UTC
Sent	17 Jul 2011, 11:58:04 UTC
Report deadline	16 Oct 2011, 19:25:15 UTC
Received	10 Feb 2012, 19:13:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1138353
Run time	9 days 4 hours 45 min
CPU time	5 days 17 hours 16 min 20 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7160, selfPID=7160, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3908, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3584, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:23:15 (3184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3712, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:25:55 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:56 (6992): No heartbeat from core client for 30 sec - exiting 13:25:57 (6992): No heartbeat from core client for 30 sec - exiting 13:25:58 (6992): No heartbeat from core client for 30 sec - exiting 13:25:59 (6992): No heartbeat from core client for 30 sec - exiting 13:26:00 (6992): No heartbeat from core client for 30 sec - exiting 13:26:01 (6992): No heartbeat from core client for 30 sec - exiting 13:26:02 (6992): No heartbeat from core client for 30 sec - exiting 13:26:03 (6992): No heartbeat from core client for 30 sec - exiting 13:26:04 (6992): No heartbeat from core client for 30 sec - exiting 13:26:05 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5156, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5652, selfPID=5652, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:27:16 (192): No heartbeat from core client for 30 sec - exiting 15:27:18 (192): No heartbeat from core client for 30 sec - exiting 15:27:20 (192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:27:30 (5932): No heartbeat from core client for 30 sec - exiting 19:27:31 (5932): No heartbeat from core client for 30 sec - exiting 19:27:32 (5932): No heartbeat from core client for 30 sec - exiting 19:27:33 (5932): No heartbeat from core client for 30 sec - exiting 19:27:34 (5932): No heartbeat from core client for 30 sec - exiting 19:27:35 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:36 (5932): No heartbeat from core client for 30 sec - exiting 19:27:37 (5932): No heartbeat from core client for 30 sec - exiting 19:27:38 (5932): No heartbeat from core client for 30 sec - exiting 19:27:39 (5932): No heartbeat from core client for 30 sec - exiting 19:27:40 (5932): No heartbeat from core client for 30 sec - exiting 19:27:41 (5932): No heartbeat from core client for 30 sec - exiting 19:27:42 (5932): No heartbeat from core client for 30 sec - exiting 19:27:44 (5932): No heartbeat from core client for 30 sec - exiting 19:27:45 (5932): No heartbeat from core client for 30 sec - exiting 19:27:46 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:05:22 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:10:54 (6320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:27:58 (6748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:51:41 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:54 (4476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:44:12 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... 12:59:14 (4516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:33:58 (4744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:14:50 (2080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:56:35 (5220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:04 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:51:56 (4328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... 22:12:33 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:11:18 (2436): No heartbeat from core client for 30 sec - exiting 12:11:20 (2436): No heartbeat from core client for 30 sec - exiting 12:11:21 (2436): No heartbeat from core client for 30 sec - exiting 12:11:22 (2436): No heartbeat from core client for 30 sec - exiting 12:11:23 (2436): No heartbeat from core client for 30 sec - exiting 12:11:24 (2436): No heartbeat from core client for 30 sec - exiting 12:11:25 (2436): No heartbeat from core client for 30 sec - exiting 12:11:26 (2436): No heartbeat from core client for 30 sec - exiting 12:11:27 (2436): No heartbeat from core client for 30 sec - exiting 12:11:28 (2436): No heartbeat from core client for 30 sec - exiting 12:11:29 (2436): No heartbeat from core client for 30 sec - exiting 12:11:30 (2436): No heartbeat from core client for 30 sec - exiting 12:11:31 (2436): No heartbeat from core client for 30 sec - exiting 12:11:32 (2436): No heartbeat from core client for 30 sec - exiting 12:11:33 (2436): No heartbeat from core client for 30 sec - exiting 12:11:34 (2436): No heartbeat from core client for 30 sec - exiting 12:11:35 (2436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:36 (2436): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:30:48 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:42:50 (940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:51 (940): No heartbeat from core client for 30 sec - exiting 22:48:13 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:36:34 (744): No heartbeat from core client for 30 sec - exiting 23:36:35 (744): No heartbeat from core client for 30 sec - exiting 23:36:36 (744): No heartbeat from core client for 30 sec - exiting 23:36:37 (744): No heartbeat from core client for 30 sec - exiting 23:36:38 (744): No heartbeat from core client for 30 sec - exiting 23:36:40 (744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:39:33 (3860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yd01ko.pja8c10 Error converting file to netcdf: dataout/yd01ko.pia8c10 Error converting file to netcdf: dataout/yd01ko.pfa8c10 Error converting file to netcdf: dataout/yd01ka.pha8c10 Error converting file to netcdf: dataout/yd01ka.pga8c10 Error converting file to netcdf: dataout/yd01ka.pea8c10 Error converting file to netcdf: dataout/yd01ka.pda8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:15:04 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:32 (3184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Dec 2011 14:51:25	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	207,360	488,858	2.3575
25 Sep 2011 19:40:02	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	181,440	428,798	2.3633
27 Aug 2011 15:11:15	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	155,520	368,440	2.3691
21 Aug 2011 13:50:58	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	129,600	306,197	2.3626
18 Aug 2011 15:21:32	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	103,680	246,484	2.3774
04 Aug 2011 17:41:57	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	77,760	184,356	2.3708
01 Aug 2011 00:04:11	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	51,840	122,933	2.3714
01 Aug 2011 00:04:11	1138353	13103281	hadcm3n_yd01_1900_40_007349707_0	25,920	61,896	2.3880