Task 13654697

Name	hadcm3n_yh2y_1900_40_007516369_2
Workunit	7713844
Created	23 Nov 2011, 1:41:47 UTC
Sent	23 Nov 2011, 1:44:54 UTC
Report deadline	22 Feb 2012, 9:12:05 UTC
Received	31 Jan 2012, 8:00:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1059549
Run time	25 days 2 hours 42 min 12 sec
CPU time	25 days 2 hours 42 min 12 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2260, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:52:28 (2284): No heartbeat from core client for 30 sec - exiting 12:52:29 (2284): No heartbeat from core client for 30 sec - exiting 12:52:30 (2284): No heartbeat from core client for 30 sec - exiting 12:52:31 (2284): No heartbeat from core client for 30 sec - exiting 12:52:32 (2284): No heartbeat from core client for 30 sec - exiting 12:52:33 (2284): No heartbeat from core client for 30 sec - exiting 12:52:34 (2284): No heartbeat from core client for 30 sec - exiting 12:52:35 (2284): No heartbeat from core client for 30 sec - exiting 12:52:36 (2284): No heartbeat from core client for 30 sec - exiting 12:52:37 (2284): No heartbeat from core client for 30 sec - exiting 12:52:38 (2284): No heartbeat from core client for 30 sec - exiting 12:52:39 (2284): No heartbeat from core client for 30 sec - exiting 12:52:40 (2284): No heartbeat from core client for 30 sec - exiting 12:52:41 (2284): No heartbeat from core client for 30 sec - exiting 12:52:42 (2284): No heartbeat from core client for 30 sec - exiting 12:52:43 (2284): No heartbeat from core client for 30 sec - exiting 12:52:44 (2284): No heartbeat from core client for 30 sec - exiting 12:52:45 (2284): No heartbeat from core client for 30 sec - exiting 12:52:46 (2284): No heartbeat from core client for 30 sec - exiting 12:52:47 (2284): No heartbeat from core client for 30 sec - exiting 12:52:48 (2284): No heartbeat from core client for 30 sec - exiting 12:52:49 (2284): No heartbeat from core client for 30 sec - exiting 12:52:50 (2284): No heartbeat from core client for 30 sec - exiting 12:52:51 (2284): No heartbeat from core client for 30 sec - exiting 12:52:52 (2284): No heartbeat from core client for 30 sec - exiting 12:52:53 (2284): No heartbeat from core client for 30 sec - exiting 12:52:54 (2284): No heartbeat from core client for 30 sec - exiting 12:52:55 (2284): No heartbeat from core client for 30 sec - exiting 12:52:56 (2284): No heartbeat from core client for 30 sec - exiting 12:52:57 (2284): No heartbeat from core client for 30 sec - exiting 12:52:58 (2284): No heartbeat from core client for 30 sec - exiting 12:52:59 (2284): No heartbeat from core client for 30 sec - exiting 12:53:00 (2284): No heartbeat from core client for 30 sec - exiting 12:53:01 (2284): No heartbeat from core client for 30 sec - exiting 12:53:02 (2284): No heartbeat from core client for 30 sec - exiting 12:53:03 (2284): No heartbeat from core client for 30 sec - exiting 12:53:04 (2284): No heartbeat from core client for 30 sec - exiting 12:53:05 (2284): No heartbeat from core client for 30 sec - exiting 12:53:06 (2284): No heartbeat from core client for 30 sec - exiting 12:53:07 (2284): No heartbeat from core client for 30 sec - exiting 12:53:08 (2284): No heartbeat from core client for 30 sec - exiting 12:53:09 (2284): No heartbeat from core client for 30 sec - exiting 12:53:10 (2284): No heartbeat from core client for 30 sec - exiting 12:53:11 (2284): No heartbeat from core client for 30 sec - exiting 12:53:12 (2284): No heartbeat from core client for 30 sec - exiting 12:53:13 (2284): No heartbeat from core client for 30 sec - exiting 12:53:14 (2284): No heartbeat from core client for 30 sec - exiting 12:53:15 (2284): No heartbeat from core client for 30 sec - exiting 12:53:16 (2284): No heartbeat from core client for 30 sec - exiting 12:53:17 (2284): No heartbeat from core client for 30 sec - exiting 12:53:18 (2284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:48 (2792): No heartbeat from core client for 30 sec - exiting 19:44:50 (2792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:51 (2792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:22:38 (3028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4276, selfPID=4276, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2872, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2692, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2068, iMonCtr=1 Model crash detected, will try to restart... 13:23:05 (3320): No heartbeat from core client for 30 sec - exiting 13:23:06 (3320): No heartbeat from core client for 30 sec - exiting 13:23:07 (3320): No heartbeat from core client for 30 sec - exiting 13:23:08 (3320): No heartbeat from core client for 30 sec - exiting 13:23:09 (3320): No heartbeat from core client for 30 sec - exiting 13:23:10 (3320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:14:15 (3796): No heartbeat from core client for 30 sec - exiting 09:14:16 (3796): No heartbeat from core client for 30 sec - exiting 09:14:17 (3796): No heartbeat from core client for 30 sec - exiting 09:14:18 (3796): No heartbeat from core client for 30 sec - exiting 09:14:19 (3796): No heartbeat from core client for 30 sec - exiting 09:14:20 (3796): No heartbeat from core client for 30 sec - exiting 09:14:21 (3796): No heartbeat from core client for 30 sec - exiting 09:14:23 (3796): No heartbeat from core client for 30 sec - exiting 09:14:24 (3796): No heartbeat from core client for 30 sec - exiting 09:14:25 (3796): No heartbeat from core client for 30 sec - exiting 09:14:26 (3796): No heartbeat from core client for 30 sec - exiting 09:14:27 (3796): No heartbeat from core client for 30 sec - exiting 09:14:28 (3796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:08:22 (1888): No heartbeat from core client for 30 sec - exiting 20:08:23 (1888): No heartbeat from core client for 30 sec - exiting 20:08:24 (1888): No heartbeat from core client for 30 sec - exiting 20:08:25 (1888): No heartbeat from core client for 30 sec - exiting 20:08:26 (1888): No heartbeat from core client for 30 sec - exiting 20:08:27 (1888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2796, iMonCtr=1 Model crash detected, will try to restart... 09:30:39 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:55:29 (5492): No heartbeat from core client for 30 sec - exiting 08:55:30 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yh2yko.pjb9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1152, iMonCtr=1 Model crash detected, will try to restart... 12:15:33 (2468): No heartbeat from core client for 30 sec - exiting 12:15:34 (2468): No heartbeat from core client for 30 sec - exiting 12:15:35 (2468): No heartbeat from core client for 30 sec - exiting 12:15:36 (2468): No heartbeat from core client for 30 sec - exiting 12:15:38 (2468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:15:39 (2468): No heartbeat from core client for 30 sec - exiting 12:20:42 (2388): No heartbeat from core client for 30 sec - exiting 12:20:43 (2388): No heartbeat from core client for 30 sec - exiting 12:20:44 (2388): No heartbeat from core client for 30 sec - exiting 12:20:45 (2388): No heartbeat from core client for 30 sec - exiting 12:20:46 (2388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4348, iMonCtr=1 Model crash detected, will try to restart... 14:52:45 (2496): No heartbeat from core client for 30 sec - exiting 14:52:46 (2496): No heartbeat from core client for 30 sec - exiting 14:52:47 (2496): No heartbeat from core client for 30 sec - exiting 14:52:48 (2496): No heartbeat from core client for 30 sec - exiting 14:52:49 (2496): No heartbeat from core client for 30 sec - exiting 14:52:50 (2496): No heartbeat from core client for 30 sec - exiting 14:52:51 (2496): No heartbeat from core client for 30 sec - exiting 14:52:52 (2496): No heartbeat from core client for 30 sec - exiting 14:52:53 (2496): No heartbeat from core client for 30 sec - exiting 14:52:54 (2496): No heartbeat from core client for 30 sec - exiting 14:52:55 (2496): No heartbeat from core client for 30 sec - exiting 14:52:56 (2496): No heartbeat from core client for 30 sec - exiting 14:52:57 (2496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:46:00 (3872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:16:19 (2540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:10 (2892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2228, iMonCtr=1 Model crash detected, will try to restart... 09:25:16 (2516): No heartbeat from core client for 30 sec - exiting 09:25:17 (2516): No heartbeat from core client for 30 sec - exiting 09:25:18 (2516): No heartbeat from core client for 30 sec - exiting 09:25:19 (2516): No heartbeat from core client for 30 sec - exiting 09:25:20 (2516): No heartbeat from core client for 30 sec - exiting 09:25:21 (2516): No heartbeat from core client for 30 sec - exiting 09:25:22 (2516): No heartbeat from core client for 30 sec - exiting 09:25:24 (2516): No heartbeat from core client for 30 sec - exiting 09:25:25 (2516): No heartbeat from core client for 30 sec - exiting 09:25:26 (2516): No heartbeat from core client for 30 sec - exiting 09:25:27 (2516): No heartbeat from core client for 30 sec - exiting 09:25:28 (2516): No heartbeat from core client for 30 sec - exiting 09:25:29 (2516): No heartbeat from core client for 30 sec - exiting 09:25:30 (2516): No heartbeat from core client for 30 sec - exiting 09:25:31 (2516): No heartbeat from core client for 30 sec - exiting 09:25:32 (2516): No heartbeat from core client for 30 sec - exiting 09:25:33 (2516): No heartbeat from core client for 30 sec - exiting 09:25:34 (2516): No heartbeat from core client for 30 sec - exiting 09:25:36 (2516): No heartbeat from core client for 30 sec - exiting 09:25:37 (2516): No heartbeat from core client for 30 sec - exiting 09:25:38 (2516): No heartbeat from core client for 30 sec - exiting 09:25:39 (2516): No heartbeat from core client for 30 sec - exiting 09:25:40 (2516): No heartbeat from core client for 30 sec - exiting 09:25:41 (2516): No heartbeat from core client for 30 sec - exiting 09:25:42 (2516): No heartbeat from core client for 30 sec - exiting 09:25:43 (2516): No heartbeat from core client for 30 sec - exiting 09:25:44 (2516): No heartbeat from core client for 30 sec - exiting 09:25:45 (2516): No heartbeat from core client for 30 sec - exiting 09:25:46 (2516): No heartbeat from core client for 30 sec - exiting 09:25:48 (2516): No heartbeat from core client for 30 sec - exiting 09:25:49 (2516): No heartbeat from core client for 30 sec - exiting 09:25:50 (2516): No heartbeat from core client for 30 sec - exiting 09:25:51 (2516): No heartbeat from core client for 30 sec - exiting 09:25:52 (2516): No heartbeat from core client for 30 sec - exiting 09:25:53 (2516): No heartbeat from core client for 30 sec - exiting 09:25:54 (2516): No heartbeat from core client for 30 sec - exiting 09:25:55 (2516): No heartbeat from core client for 30 sec - exiting 09:25:56 (2516): No heartbeat from core client for 30 sec - exiting 09:25:57 (2516): No heartbeat from core client for 30 sec - exiting 09:25:58 (2516): No heartbeat from core client for 30 sec - exiting 09:25:59 (2516): No heartbeat from core client for 30 sec - exiting 09:26:00 (2516): No heartbeat from core client for 30 sec - exiting 09:26:01 (2516): No heartbeat from core client for 30 sec - exiting 09:26:02 (2516): No heartbeat from core client for 30 sec - exiting 09:26:03 (2516): No heartbeat from core client for 30 sec - exiting 09:26:04 (2516): No heartbeat from core client for 30 sec - exiting 09:26:05 (2516): No heartbeat from core client for 30 sec - exiting 09:26:06 (2516): No heartbeat from core client for 30 sec - exiting 09:26:07 (2516): No heartbeat from core client for 30 sec - exiting 09:26:08 (2516): No heartbeat from core client for 30 sec - exiting 09:26:09 (2516): No heartbeat from core client for 30 sec - exiting 09:26:10 (2516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:05:04 (2404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=1 Model crash detected, will try to restart... 08:12:23 (2412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Jan 2012 23:27:33	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	1,036,800	2,169,707	2.0927
27 Jan 2012 08:52:39	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	1,010,880	2,117,422	2.0946
26 Jan 2012 05:13:29	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	984,960	2,065,204	2.0967
25 Jan 2012 01:33:53	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	959,040	2,012,763	2.0987
22 Jan 2012 23:48:13	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	933,120	1,960,807	2.1013
21 Jan 2012 10:26:07	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	907,200	1,908,695	2.1039
20 Jan 2012 09:04:52	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	881,280	1,856,861	2.1070
19 Jan 2012 02:30:15	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	855,360	1,804,989	2.1102
17 Jan 2012 23:57:03	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	829,440	1,752,964	2.1134
16 Jan 2012 06:49:12	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	803,520	1,699,411	2.1150
14 Jan 2012 08:37:07	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	777,600	1,645,890	2.1166
13 Jan 2012 06:22:09	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	751,680	1,590,916	2.1165
11 Jan 2012 05:23:10	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	725,760	1,536,042	2.1165
09 Jan 2012 07:48:04	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	699,840	1,482,286	2.1180
07 Jan 2012 22:27:37	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	673,920	1,427,614	2.1184
04 Jan 2012 07:35:58	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	648,000	1,374,362	2.1209
02 Jan 2012 10:30:12	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	622,080	1,320,056	2.1220
01 Jan 2012 07:17:02	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	596,160	1,265,491	2.1227
31 Dec 2011 00:09:04	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	570,240	1,211,195	2.1240
29 Dec 2011 11:24:41	1059549	13654697	hadcm3n_yh2y_1900_40_007516369_2	544,320	1,156,816	2.1252