Task 15272939

Name	hadcm3n_zg32_1880_40_008198151_0
Workunit	8353275
Created	12 Sep 2012, 16:13:51 UTC
Sent	15 Sep 2012, 14:16:04 UTC
Report deadline	15 Dec 2012, 21:43:15 UTC
Received	18 Nov 2012, 20:06:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1210168
Run time	9 days 15 hours 41 min 9 sec
CPU time	9 days 10 hours 9 min 48 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	2.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 18:38:09 (4372): No heartbeat from core client for 30 sec - exiting 18:38:10 (4372): No heartbeat from core client for 30 sec - exiting 18:38:11 (4372): No heartbeat from core client for 30 sec - exiting 18:38:12 (4372): No heartbeat from core client for 30 sec - exiting 18:38:13 (4372): No heartbeat from core client for 30 sec - exiting 18:38:14 (4372): No heartbeat from core client for 30 sec - exiting 18:38:15 (4372): No heartbeat from core client for 30 sec - exiting 18:38:16 (4372): No heartbeat from core client for 30 sec - exiting 18:38:17 (4372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4476, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4476, iMonCtr=1 Model crash detected, will try to restart... 18:02:37 (4424): No heartbeat from core client for 30 sec - exiting 18:02:38 (4424): No heartbeat from core client for 30 sec - exiting 18:02:39 (4424): No heartbeat from core client for 30 sec - exiting 18:02:40 (4424): No heartbeat from core client for 30 sec - exiting 18:02:41 (4424): No heartbeat from core client for 30 sec - exiting 18:02:42 (4424): No heartbeat from core client for 30 sec - exiting 18:02:43 (4424): No heartbeat from core client for 30 sec - exiting 18:02:44 (4424): No heartbeat from core client for 30 sec - exiting 18:02:45 (4424): No heartbeat from core client for 30 sec - exiting 18:02:46 (4424): No heartbeat from core client for 30 sec - exiting 18:02:47 (4424): No heartbeat from core client for 30 sec - exiting 18:02:48 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:55:13 (4360): No heartbeat from core client for 30 sec - exiting 19:55:14 (4360): No heartbeat from core client for 30 sec - exiting 19:55:15 (4360): No heartbeat from core client for 30 sec - exiting 19:55:16 (4360): No heartbeat from core client for 30 sec - exiting 19:55:17 (4360): No heartbeat from core client for 30 sec - exiting 19:55:18 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:07 (4896): No heartbeat from core client for 30 sec - exiting 12:53:08 (4896): No heartbeat from core client for 30 sec - exiting 12:53:09 (4896): No heartbeat from core client for 30 sec - exiting 12:53:10 (4896): No heartbeat from core client for 30 sec - exiting 12:53:11 (4896): No heartbeat from core client for 30 sec - exiting 12:53:13 (4896): No heartbeat from core client for 30 sec - exiting 12:53:14 (4896): No heartbeat from core client for 30 sec - exiting 12:53:15 (4896): No heartbeat from core client for 30 sec - exiting 12:53:16 (4896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:55:43 (4156): No heartbeat from core client for 30 sec - exiting 10:55:44 (4156): No heartbeat from core client for 30 sec - exiting 10:55:45 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:38:30 (4376): No heartbeat from core client for 30 sec - exiting 19:38:31 (4376): No heartbeat from core client for 30 sec - exiting 19:38:32 (4376): No heartbeat from core client for 30 sec - exiting 19:38:33 (4376): No heartbeat from core client for 30 sec - exiting 19:38:34 (4376): No heartbeat from core client for 30 sec - exiting 19:38:35 (4376): No heartbeat from core client for 30 sec - exiting 19:38:36 (4376): No heartbeat from core client for 30 sec - exiting 19:38:37 (4376): No heartbeat from core client for 30 sec - exiting 19:38:38 (4376): No heartbeat from core client for 30 sec - exiting 19:38:39 (4376): No heartbeat from core client for 30 sec - exiting 19:38:40 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:41 (4376): No heartbeat from core client for 30 sec - exiting 19:38:42 (4376): No heartbeat from core client for 30 sec - exiting 19:38:43 (4376): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4296, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:10:34 (3308): No heartbeat from core client for 30 sec - exiting 19:10:36 (3308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:58:35 (6508): No heartbeat from core client for 30 sec - exiting 18:58:36 (6508): No heartbeat from core client for 30 sec - exiting 18:58:37 (6508): No heartbeat from core client for 30 sec - exiting 18:58:38 (6508): No heartbeat from core client for 30 sec - exiting 18:58:39 (6508): No heartbeat from core client for 30 sec - exiting 18:58:40 (6508): No heartbeat from core client for 30 sec - exiting 18:58:41 (6508): No heartbeat from core client for 30 sec - exiting 18:58:42 (6508): No heartbeat from core client for 30 sec - exiting 18:58:43 (6508): No heartbeat from core client for 30 sec - exiting 18:58:44 (6508): No heartbeat from core client for 30 sec - exiting 18:58:45 (6508): No heartbeat from core client for 30 sec - exiting 18:58:46 (6508): No heartbeat from core client for 30 sec - exiting 18:58:47 (6508): No heartbeat from core client for 30 sec - exiting 18:58:48 (6508): No heartbeat from core client for 30 sec - exiting 18:58:49 (6508): No heartbeat from core client for 30 sec - exiting 18:58:50 (6508): No heartbeat from core client for 30 sec - exiting 18:58:51 (6508): No heartbeat from core client for 30 sec - exiting 18:58:52 (6508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:33:38 (5072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:21:03 (4424): No heartbeat from core client for 30 sec - exiting 18:21:04 (4424): No heartbeat from core client for 30 sec - exiting 18:21:06 (4424): No heartbeat from core client for 30 sec - exiting 18:21:07 (4424): No heartbeat from core client for 30 sec - exiting 18:21:08 (4424): No heartbeat from core client for 30 sec - exiting 18:21:09 (4424): No heartbeat from core client for 30 sec - exiting 18:21:10 (4424): No heartbeat from core client for 30 sec - exiting 18:21:11 (4424): No heartbeat from core client for 30 sec - exiting 18:21:12 (4424): No heartbeat from core client for 30 sec - exiting 18:21:13 (4424): No heartbeat from core client for 30 sec - exiting 18:21:14 (4424): No heartbeat from core client for 30 sec - exiting 18:21:15 (4424): No heartbeat from core client for 30 sec - exiting 18:21:16 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Nov 2012 16:05:19	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	881,280	793,337	0.9002
13 Nov 2012 19:23:21	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	855,360	770,043	0.9003
11 Nov 2012 17:18:32	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	829,440	746,817	0.9004
11 Nov 2012 09:32:44	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	803,520	723,075	0.8999
08 Nov 2012 21:48:28	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	777,600	699,492	0.8996
08 Nov 2012 15:50:37	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	751,680	676,078	0.8994
06 Nov 2012 15:26:33	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	725,760	652,745	0.8994
05 Nov 2012 21:03:29	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	699,840	629,508	0.8995
04 Nov 2012 12:41:10	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	673,920	605,946	0.8991
31 Oct 2012 19:32:09	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	648,000	582,393	0.8988
29 Oct 2012 18:28:04	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	622,080	559,099	0.8988
28 Oct 2012 13:14:24	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	596,160	535,553	0.8983
27 Oct 2012 15:07:01	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	570,240	512,123	0.8981
25 Oct 2012 19:36:36	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	544,320	488,825	0.8980
22 Oct 2012 20:34:50	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	518,400	465,487	0.8979
21 Oct 2012 11:50:18	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	492,480	441,974	0.8974
20 Oct 2012 12:37:00	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	466,560	418,540	0.8971
19 Oct 2012 18:40:41	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	440,640	395,130	0.8967
18 Oct 2012 17:58:05	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	414,720	371,531	0.8959
12 Oct 2012 16:42:03	1210168	15272939	hadcm3n_zg32_1880_40_008198151_0	388,800	348,065	0.8952