Task 13117782

Name	hadcm3n_yild_1900_40_007356955_1
Workunit	7554385
Created	6 Jul 2011, 14:49:56 UTC
Sent	9 Jul 2011, 6:37:50 UTC
Report deadline	8 Oct 2011, 14:05:01 UTC
Received	30 Aug 2011, 2:49:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1125581
Run time	17 days 18 hours 51 min 26 sec
CPU time	15 days 14 hours 29 min 4 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7104, iMonCtr=1 Model crash detected, will try to restart... 06:36:55 (5520): No heartbeat from core client for 30 sec - exiting 06:36:56 (5520): No heartbeat from core client for 30 sec - exiting 06:36:57 (5520): No heartbeat from core client for 30 sec - exiting 06:36:58 (5520): No heartbeat from core client for 30 sec - exiting 06:36:59 (5520): No heartbeat from core client for 30 sec - exiting 06:37:00 (5520): No heartbeat from core client for 30 sec - exiting 06:37:01 (5520): No heartbeat from core client for 30 sec - exiting 06:37:02 (5520): No heartbeat from core client for 30 sec - exiting 06:37:03 (5520): No heartbeat from core client for 30 sec - exiting 06:37:04 (5520): No heartbeat from core client for 30 sec - exiting 06:37:05 (5520): No heartbeat from core client for 30 sec - exiting 06:37:06 (5520): No heartbeat from core client for 30 sec - exiting 06:37:07 (5520): No heartbeat from core client for 30 sec - exiting 06:37:08 (5520): No heartbeat from core client for 30 sec - exiting 06:37:09 (5520): No heartbeat from core client for 30 sec - exiting 06:37:10 (5520): No heartbeat from core client for 30 sec - exiting 06:37:11 (5520): No heartbeat from core client for 30 sec - exiting 06:37:12 (5520): No heartbeat from core client for 30 sec - exiting 06:37:13 (5520): No heartbeat from core client for 30 sec - exiting 06:37:14 (5520): No heartbeat from core client for 30 sec - exiting 06:37:15 (5520): No heartbeat from core client for 30 sec - exiting 06:37:16 (5520): No heartbeat from core client for 30 sec - exiting 06:37:17 (5520): No heartbeat from core client for 30 sec - exiting 06:37:18 (5520): No heartbeat from core client for 30 sec - exiting 06:37:19 (5520): No heartbeat from core client for 30 sec - exiting 06:37:20 (5520): No heartbeat from core client for 30 sec - exiting 06:37:21 (5520): No heartbeat from core client for 30 sec - exiting 06:37:22 (5520): No heartbeat from core client for 30 sec - exiting 06:37:23 (5520): No heartbeat from core client for 30 sec - exiting 06:37:24 (5520): No heartbeat from core client for 30 sec - exiting 06:37:25 (5520): No heartbeat from core client for 30 sec - exiting 06:37:26 (5520): No heartbeat from core client for 30 sec - exiting 06:37:27 (5520): No heartbeat from core client for 30 sec - exiting 06:37:28 (5520): No heartbeat from core client for 30 sec - exiting 06:37:29 (5520): No heartbeat from core client for 30 sec - exiting 06:37:30 (5520): No heartbeat from core client for 30 sec - exiting 06:37:31 (5520): No heartbeat from core client for 30 sec - exiting 06:37:32 (5520): No heartbeat from core client for 30 sec - exiting 06:37:33 (5520): No heartbeat from core client for 30 sec - exiting 06:37:34 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:37:35 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:06:57 (6964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8284, iMonCtr=1 Model crash detected, will try to restart... 07:49:54 (6260): No heartbeat from core client for 30 sec - exiting 07:49:55 (6260): No heartbeat from core client for 30 sec - exiting 07:49:56 (6260): No heartbeat from core client for 30 sec - exiting 07:49:57 (6260): No heartbeat from core client for 30 sec - exiting 07:49:58 (6260): No heartbeat from core client for 30 sec - exiting 07:50:00 (6260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:36:04 (7716): No heartbeat from core client for 30 sec - exiting 19:36:05 (7716): No heartbeat from core client for 30 sec - exiting 19:36:06 (7716): No heartbeat from core client for 30 sec - exiting 19:36:07 (7716): No heartbeat from core client for 30 sec - exiting 19:36:08 (7716): No heartbeat from core client for 30 sec - exiting 19:36:09 (7716): No heartbeat from core client for 30 sec - exiting 19:36:10 (7716): No heartbeat from core client for 30 sec - exiting 19:36:11 (7716): No heartbeat from core client for 30 sec - exiting 19:36:12 (7716): No heartbeat from core client for 30 sec - exiting 19:36:13 (7716): No heartbeat from core client for 30 sec - exiting 19:36:14 (7716): No heartbeat from core client for 30 sec - exiting 19:36:15 (7716): No heartbeat from core client for 30 sec - exiting 19:36:16 (7716): No heartbeat from core client for 30 sec - exiting 19:36:17 (7716): No heartbeat from core client for 30 sec - exiting 19:36:18 (7716): No heartbeat from core client for 30 sec - exiting 19:36:19 (7716): No heartbeat from core client for 30 sec - exiting 19:36:20 (7716): No heartbeat from core client for 30 sec - exiting 19:36:21 (7716): No heartbeat from core client for 30 sec - exiting 19:36:22 (7716): No heartbeat from core client for 30 sec - exiting 19:36:23 (7716): No heartbeat from core client for 30 sec - exiting 19:36:24 (7716): No heartbeat from core client for 30 sec - exiting 19:36:25 (7716): No heartbeat from core client for 30 sec - exiting 19:36:26 (7716): No heartbeat from core client for 30 sec - exiting 19:36:27 (7716): No heartbeat from core client for 30 sec - exiting 19:36:28 (7716): No heartbeat from core client for 30 sec - exiting 19:36:29 (7716): No heartbeat from core client for 30 sec - exiting 19:36:30 (7716): No heartbeat from core client for 30 sec - exiting 19:36:31 (7716): No heartbeat from core client for 30 sec - exiting 19:36:32 (7716): No heartbeat from core client for 30 sec - exiting 19:36:33 (7716): No heartbeat from core client for 30 sec - exiting 19:36:34 (7716): No heartbeat from core client for 30 sec - exiting 19:36:35 (7716): No heartbeat from core client for 30 sec - exiting 19:36:36 (7716): No heartbeat from core client for 30 sec - exiting 19:36:37 (7716): No heartbeat from core client for 30 sec - exiting 19:36:38 (7716): No heartbeat from core client for 30 sec - exiting 19:36:39 (7716): No heartbeat from core client for 30 sec - exiting 19:36:40 (7716): No heartbeat from core client for 30 sec - exiting 19:36:41 (7716): No heartbeat from core client for 30 sec - exiting 19:36:42 (7716): No heartbeat from core client for 30 sec - exiting 19:36:43 (7716): No heartbeat from core client for 30 sec - exiting 19:36:44 (7716): No heartbeat from core client for 30 sec - exiting 19:36:45 (7716): No heartbeat from core client for 30 sec - exiting 19:36:46 (7716): No heartbeat from core client for 30 sec - exiting 19:36:47 (7716): No heartbeat from core client for 30 sec - exiting 19:36:48 (7716): No heartbeat from core client for 30 sec - exiting 19:36:49 (7716): No heartbeat from core client for 30 sec - exiting 19:36:50 (7716): No heartbeat from core client for 30 sec - exiting 19:36:51 (7716): No heartbeat from core client for 30 sec - exiting 19:36:52 (7716): No heartbeat from core client for 30 sec - exiting 19:36:53 (7716): No heartbeat from core client for 30 sec - exiting 19:36:54 (7716): No heartbeat from core client for 30 sec - exiting 19:36:55 (7716): No heartbeat from core client for 30 sec - exiting 19:36:56 (7716): No heartbeat from core client for 30 sec - exiting 19:36:57 (7716): No heartbeat from core client for 30 sec - exiting 19:36:58 (7716): No heartbeat from core client for 30 sec - exiting 19:36:59 (7716): No heartbeat from core client for 30 sec - exiting 19:37:00 (7716): No heartbeat from core client for 30 sec - exiting 19:37:01 (7716): No heartbeat from core client for 30 sec - exiting 19:37:02 (7716): No heartbeat from core client for 30 sec - exiting 19:37:03 (7716): No heartbeat from core client for 30 sec - exiting 19:37:04 (7716): No heartbeat from core client for 30 sec - exiting 19:37:05 (7716): No heartbeat from core client for 30 sec - exiting 19:37:06 (7716): No heartbeat from core client for 30 sec - exiting 19:37:07 (7716): No heartbeat from core client for 30 sec - exiting 19:37:08 (7716): No heartbeat from core client for 30 sec - exiting 19:37:09 (7716): No heartbeat from core client for 30 sec - exiting 19:37:10 (7716): No heartbeat from core client for 30 sec - exiting 19:37:11 (7716): No heartbeat from core client for 30 sec - exiting 19:37:12 (7716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1828, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:46:33 (14932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11508, iMonCtr=1 Model crash detected, will try to restart... 08:10:25 (6280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Aug 2011 02:50:10	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	777,600	1,348,141	1.7337
28 Aug 2011 12:26:58	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	751,680	1,305,390	1.7366
27 Aug 2011 22:36:30	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	725,760	1,262,607	1.7397
26 Aug 2011 21:37:12	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	699,840	1,219,796	1.7430
26 Aug 2011 08:12:45	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	673,920	1,177,147	1.7467
25 Aug 2011 03:19:33	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	648,000	1,133,386	1.7491
24 Aug 2011 02:07:24	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	622,080	1,088,637	1.7500
23 Aug 2011 08:29:57	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	596,160	1,044,412	1.7519
22 Aug 2011 15:19:46	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	570,240	999,626	1.7530
21 Aug 2011 23:47:14	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	544,320	955,235	1.7549
21 Aug 2011 09:44:39	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	518,400	910,697	1.7567
20 Aug 2011 06:25:01	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	492,480	866,649	1.7598
19 Aug 2011 08:54:16	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	466,560	822,918	1.7638
18 Aug 2011 11:56:20	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	440,640	778,748	1.7673
17 Aug 2011 07:21:33	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	414,720	735,457	1.7734
16 Aug 2011 04:05:26	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	388,800	691,574	1.7787
15 Aug 2011 00:33:56	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	362,880	648,478	1.7870
27 Jul 2011 03:07:43	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	336,960	603,343	1.7905
26 Jul 2011 09:18:12	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	311,040	556,562	1.7894
25 Jul 2011 23:03:33	1125581	13117782	hadcm3n_yild_1900_40_007356955_1	285,120	510,425	1.7902