Task 15891762

Name	hadcm3n_o4uq_1980_40_008386526_3
Workunit	8537385
Created	11 Jul 2013, 3:47:55 UTC
Sent	11 Jul 2013, 5:09:04 UTC
Report deadline	10 Oct 2013, 12:36:15 UTC
Received	24 Aug 2013, 2:09:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1181321
Run time	28 days 20 hours 2 min 12 sec
CPU time	23 days 16 hours 43 min 1 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	2.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 08:47:48 (2216): No heartbeat from core client for 30 sec - exiting 08:47:49 (2216): No heartbeat from core client for 30 sec - exiting 08:47:50 (2216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3328, iMonCtr=1 Model crash detected, will try to restart... 19:25:23 (4436): No heartbeat from core client for 30 sec - exiting 19:25:24 (4436): No heartbeat from core client for 30 sec - exiting 19:25:25 (4436): No heartbeat from core client for 30 sec - exiting 19:25:26 (4436): No heartbeat from core client for 30 sec - exiting 19:25:27 (4436): No heartbeat from core client for 30 sec - exiting 19:25:28 (4436): No heartbeat from core client for 30 sec - exiting 19:25:29 (4436): No heartbeat from core client for 30 sec - exiting 19:25:30 (4436): No heartbeat from core client for 30 sec - exiting 19:25:31 (4436): No heartbeat from core client for 30 sec - exiting 19:25:32 (4436): No heartbeat from core client for 30 sec - exiting 19:25:33 (4436): No heartbeat from core client for 30 sec - exiting 19:25:34 (4436): No heartbeat from core client for 30 sec - exiting 19:25:35 (4436): No heartbeat from core client for 30 sec - exiting 19:25:36 (4436): No heartbeat from core client for 30 sec - exiting 19:25:37 (4436): No heartbeat from core client for 30 sec - exiting 19:25:38 (4436): No heartbeat from core client for 30 sec - exiting 19:25:39 (4436): No heartbeat from core client for 30 sec - exiting 19:25:40 (4436): No heartbeat from core client for 30 sec - exiting 19:25:41 (4436): No heartbeat from core client for 30 sec - exiting 19:25:42 (4436): No heartbeat from core client for 30 sec - exiting 19:25:43 (4436): No heartbeat from core client for 30 sec - exiting 19:25:44 (4436): No heartbeat from core client for 30 sec - exiting 19:25:45 (4436): No heartbeat from core client for 30 sec - exiting 19:25:46 (4436): No heartbeat from core client for 30 sec - exiting 19:25:47 (4436): No heartbeat from core client for 30 sec - exiting 19:25:48 (4436): No heartbeat from core client for 30 sec - exiting 19:25:49 (4436): No heartbeat from core client for 30 sec - exiting 19:25:50 (4436): No heartbeat from core client for 30 sec - exiting 19:25:51 (4436): No heartbeat from core client for 30 sec - exiting 19:25:52 (4436): No heartbeat from core client for 30 sec - exiting 19:25:53 (4436): No heartbeat from core client for 30 sec - exiting 19:25:54 (4436): No heartbeat from core client for 30 sec - exiting 19:25:55 (4436): No heartbeat from core client for 30 sec - exiting 19:25:56 (4436): No heartbeat from core client for 30 sec - exiting 19:25:57 (4436): No heartbeat from core client for 30 sec - exiting 19:25:58 (4436): No heartbeat from core client for 30 sec - exiting 19:25:59 (4436): No heartbeat from core client for 30 sec - exiting 19:26:00 (4436): No heartbeat from core client for 30 sec - exiting 19:26:01 (4436): No heartbeat from core client for 30 sec - exiting 19:26:02 (4436): No heartbeat from core client for 30 sec - exiting 19:26:03 (4436): No heartbeat from core client for 30 sec - exiting 19:26:04 (4436): No heartbeat from core client for 30 sec - exiting 19:26:05 (4436): No heartbeat from core client for 30 sec - exiting 19:26:06 (4436): No heartbeat from core client for 30 sec - exiting 19:26:07 (4436): No heartbeat from core client for 30 sec - exiting 19:26:08 (4436): No heartbeat from core client for 30 sec - exiting 19:26:09 (4436): No heartbeat from core client for 30 sec - exiting 19:26:10 (4436): No heartbeat from core client for 30 sec - exiting 19:26:11 (4436): No heartbeat from core client for 30 sec - exiting 19:26:12 (4436): No heartbeat from core client for 30 sec - exiting 19:26:13 (4436): No heartbeat from core client for 30 sec - exiting 19:26:14 (4436): No heartbeat from core client for 30 sec - exiting 19:26:15 (4436): No heartbeat from core client for 30 sec - exiting 19:26:16 (4436): No heartbeat from core client for 30 sec - exiting 19:26:17 (4436): No heartbeat from core client for 30 sec - exiting 19:26:18 (4436): No heartbeat from core client for 30 sec - exiting 19:26:19 (4436): No heartbeat from core client for 30 sec - exiting 19:26:20 (4436): No heartbeat from core client for 30 sec - exiting 19:26:21 (4436): No heartbeat from core client for 30 sec - exiting 19:26:22 (4436): No heartbeat from core client for 30 sec - exiting 19:26:23 (4436): No heartbeat from core client for 30 sec - exiting 19:26:24 (4436): No heartbeat from core client for 30 sec - exiting 19:26:25 (4436): No heartbeat from core client for 30 sec - exiting 19:26:26 (4436): No heartbeat from core client for 30 sec - exiting 19:26:27 (4436): No heartbeat from core client for 30 sec - exiting 19:26:28 (4436): No heartbeat from core client for 30 sec - exiting 19:26:29 (4436): No heartbeat from core client for 30 sec - exiting 19:26:30 (4436): No heartbeat from core client for 30 sec - exiting 19:26:31 (4436): No heartbeat from core client for 30 sec - exiting 19:26:32 (4436): No heartbeat from core client for 30 sec - exiting 19:26:33 (4436): No heartbeat from core client for 30 sec - exiting 19:26:34 (4436): No heartbeat from core client for 30 sec - exiting 19:26:35 (4436): No heartbeat from core client for 30 sec - exiting 19:26:36 (4436): No heartbeat from core client for 30 sec - exiting 19:26:37 (4436): No heartbeat from core client for 30 sec - exiting 19:26:38 (4436): No heartbeat from core client for 30 sec - exiting 19:26:39 (4436): No heartbeat from core client for 30 sec - exiting 19:26:40 (4436): No heartbeat from core client for 30 sec - exiting 19:26:41 (4436): No heartbeat from core client for 30 sec - exiting 19:26:42 (4436): No heartbeat from core client for 30 sec - exiting 19:26:43 (4436): No heartbeat from core client for 30 sec - exiting 19:26:44 (4436): No heartbeat from core client for 30 sec - exiting 19:26:45 (4436): No heartbeat from core client for 30 sec - exiting 19:26:46 (4436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:28 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:57 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:06:03 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:09 (5952): No heartbeat from core client for 30 sec - exiting 12:06:10 (5952): No heartbeat from core client for 30 sec - exiting 12:06:11 (5952): No heartbeat from core client for 30 sec - exiting 12:06:12 (5952): No heartbeat from core client for 30 sec - exiting 12:06:13 (5952): No heartbeat from core client for 30 sec - exiting 12:06:14 (5952): No heartbeat from core client for 30 sec - exiting 12:06:15 (5952): No heartbeat from core client for 30 sec - exiting 12:06:16 (5952): No heartbeat from core client for 30 sec - exiting 12:06:17 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:16:31 (5636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:24:49 (3504): No heartbeat from core client for 30 sec - exiting 07:24:51 (3504): No heartbeat from core client for 30 sec - exiting 07:24:52 (3504): No heartbeat from core client for 30 sec - exiting 07:24:53 (3504): No heartbeat from core client for 30 sec - exiting 07:24:54 (3504): No heartbeat from core client for 30 sec - exiting 07:24:55 (3504): No heartbeat from core client for 30 sec - exiting 07:24:56 (3504): No heartbeat from core client for 30 sec - exiting 07:24:57 (3504): No heartbeat from core client for 30 sec - exiting 07:24:58 (3504): No heartbeat from core client for 30 sec - exiting 07:24:59 (3504): No heartbeat from core client for 30 sec - exiting 07:25:00 (3504): No heartbeat from core client for 30 sec - exiting 07:25:01 (3504): No heartbeat from core client for 30 sec - exiting 07:25:02 (3504): No heartbeat from core client for 30 sec - exiting 07:25:03 (3504): No heartbeat from core client for 30 sec - exiting 07:25:04 (3504): No heartbeat from core client for 30 sec - exiting 07:25:05 (3504): No heartbeat from core client for 30 sec - exiting 07:25:06 (3504): No heartbeat from core client for 30 sec - exiting 07:25:07 (3504): No heartbeat from core client for 30 sec - exiting 07:25:08 (3504): No heartbeat from core client for 30 sec - exiting 07:25:09 (3504): No heartbeat from core client for 30 sec - exiting 07:25:10 (3504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:05:45 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:47 (5004): No heartbeat from core client for 30 sec - exiting 03:00:58 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:36 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:38 (5400): No heartbeat from core client for 30 sec - exiting 02:14:39 (5400): No heartbeat from core client for 30 sec - exiting 02:14:40 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:45:15 (5024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:17 (5024): No heartbeat from core client for 30 sec - exiting 09:50:22 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:24 (5516): No heartbeat from core client for 30 sec - exiting 09:50:25 (5516): No heartbeat from core client for 30 sec - exiting 09:50:26 (5516): No heartbeat from core client for 30 sec - exiting 09:50:27 (5516): No heartbeat from core client for 30 sec - exiting 09:50:28 (5516): No heartbeat from core client for 30 sec - exiting 09:50:29 (5516): No heartbeat from core client for 30 sec - exiting 09:50:30 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:07:24 (4716): No heartbeat from core client for 30 sec - exiting 21:07:25 (4716): No heartbeat from core client for 30 sec - exiting 21:07:26 (4716): No heartbeat from core client for 30 sec - exiting 21:07:27 (4716): No heartbeat from core client for 30 sec - exiting 21:07:28 (4716): No heartbeat from core client for 30 sec - exiting 21:07:29 (4716): No heartbeat from core client for 30 sec - exiting 21:07:30 (4716): No heartbeat from core client for 30 sec - exiting 21:07:31 (4716): No heartbeat from core client for 30 sec - exiting 21:07:32 (4716): No heartbeat from core client for 30 sec - exiting 21:07:33 (4716): No heartbeat from core client for 30 sec - exiting 21:07:34 (4716): No heartbeat from core client for 30 sec - exiting 21:07:35 (4716): No heartbeat from core client for 30 sec - exiting 21:07:36 (4716): No heartbeat from core client for 30 sec - exiting 21:07:37 (4716): No heartbeat from core client for 30 sec - exiting 21:07:38 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:23:37 (1348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:53:05 (4904): No heartbeat from core client for 30 sec - exiting 10:53:06 (4904): No heartbeat from core client for 30 sec - exiting 10:53:07 (4904): No heartbeat from core client for 30 sec - exiting 10:53:08 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:49:43 (3564): No heartbeat from core client for 30 sec - exiting 11:49:45 (3564): No heartbeat from core client for 30 sec - exiting 11:49:46 (3564): No heartbeat from core client for 30 sec - exiting 11:49:47 (3564): No heartbeat from core client for 30 sec - exiting 11:49:48 (3564): No heartbeat from core client for 30 sec - exiting 11:49:49 (3564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:49:50 (3564): No heartbeat from core client for 30 sec - exiting 11:49:51 (3564): No heartbeat from core client for 30 sec - exiting 11:49:52 (3564): No heartbeat from core client for 30 sec - exiting 11:49:53 (3564): No heartbeat from core client for 30 sec - exiting 11:49:54 (3564): No heartbeat from core client for 30 sec - exiting 11:49:55 (3564): No heartbeat from core client for 30 sec - exiting 11:49:56 (3564): No heartbeat from core client for 30 sec - exiting 11:49:57 (3564): No heartbeat from core client for 30 sec - exiting 11:49:58 (3564): No heartbeat from core client for 30 sec - exiting 11:49:59 (3564): No heartbeat from core client for 30 sec - exiting 12:36:01 (1324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:16:27 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:29 (3756): No heartbeat from core client for 30 sec - exiting 06:16:30 (3756): No heartbeat from core client for 30 sec - exiting 06:17:49 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:06:59 (4580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:46 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:49 (5400): No heartbeat from core client for 30 sec - exiting 08:07:50 (5400): No heartbeat from core client for 30 sec - exiting 20:21:02 (2532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=1 Model crash detected, will try to restart... 02:07:05 (3728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:17:35 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:17:36 (3824): No heartbeat from core client for 30 sec - exiting 06:18:38 (3736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:39 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:41 (4684): No heartbeat from core client for 30 sec - exiting 20:00:42 (4684): No heartbeat from core client for 30 sec - exiting 20:00:43 (4684): No heartbeat from core client for 30 sec - exiting 20:00:44 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:58:12 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:13 (2444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:05 (1904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:07 (1904): No heartbeat from core client for 30 sec - exiting 16:23:23 (3212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:26 (3212): No heartbeat from core client for 30 sec - exiting 20:27:16 (3768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:27:18 (3768): No heartbeat from core client for 30 sec - exiting 20:28:07 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:32 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:34 (4124): No heartbeat from core client for 30 sec - exiting 03:06:35 (4124): No heartbeat from core client for 30 sec - exiting 03:10:16 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:32 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:34 (3180): No heartbeat from core client for 30 sec - exiting 16:43:35 (3180): No heartbeat from core client for 30 sec - exiting 16:43:36 (3180): No heartbeat from core client for 30 sec - exiting C </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Aug 2013 19:14:03	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	959,040	2,027,094	2.1137
23 Aug 2013 10:32:58	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	933,120	1,975,351	2.1169
22 Aug 2013 10:04:55	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	907,200	1,923,928	2.1207
21 Aug 2013 17:28:41	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	881,280	1,872,272	2.1245
21 Aug 2013 01:49:44	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	855,360	1,820,851	2.1288
20 Aug 2013 10:36:35	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	829,440	1,771,978	2.1364
19 Aug 2013 13:15:37	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	803,520	1,723,241	2.1446
18 Aug 2013 20:52:08	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	777,600	1,670,901	2.1488
18 Aug 2013 01:23:09	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	751,680	1,614,836	2.1483
17 Aug 2013 04:43:54	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	725,760	1,557,827	2.1465
16 Aug 2013 08:14:30	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	699,840	1,500,754	2.1444
15 Aug 2013 12:47:06	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	673,920	1,443,894	2.1425
14 Aug 2013 17:32:54	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	648,000	1,387,712	2.1415
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	622,080	1,340,910	2.1555
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	596,160	1,291,839	2.1669
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	570,240	1,239,656	2.1739
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	544,320	1,185,455	2.1779
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	518,400	1,133,115	2.1858
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	492,480	1,077,458	2.1878
14 Aug 2013 15:37:14	1181321	15891762	hadcm3n_o4uq_1980_40_008386526_3	466,560	1,022,483	2.1915