Task 13415452

Name	hadcm3n_t4b7_1940_40_007443762_4
Workunit	7641265
Created	23 Sep 2011, 13:00:18 UTC
Sent	23 Sep 2011, 13:11:17 UTC
Report deadline	23 Dec 2011, 20:38:28 UTC
Received	25 Oct 2011, 15:37:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1149830
Run time	10 days 2 hours 21 min 40 sec
CPU time	6 days 16 hours 30 min 26 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5972, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5512, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:32:04 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:16:28 (4500): No heartbeat from core client for 30 sec - exiting 20:16:30 (4500): No heartbeat from core client for 30 sec - exiting 20:16:31 (4500): No heartbeat from core client for 30 sec - exiting 20:16:32 (4500): No heartbeat from core client for 30 sec - exiting 20:16:33 (4500): No heartbeat from core client for 30 sec - exiting 20:16:34 (4500): No heartbeat from core client for 30 sec - exiting 20:16:35 (4500): No heartbeat from core client for 30 sec - exiting 20:16:36 (4500): No heartbeat from core client for 30 sec - exiting 20:16:37 (4500): No heartbeat from core client for 30 sec - exiting 20:16:38 (4500): No heartbeat from core client for 30 sec - exiting 20:16:40 (4500): No heartbeat from core client for 30 sec - exiting 20:16:41 (4500): No heartbeat from core client for 30 sec - exiting 20:16:42 (4500): No heartbeat from core client for 30 sec - exiting 20:16:43 (4500): No heartbeat from core client for 30 sec - exiting 20:16:44 (4500): No heartbeat from core client for 30 sec - exiting 20:16:45 (4500): No heartbeat from core client for 30 sec - exiting 20:16:46 (4500): No heartbeat from core client for 30 sec - exiting 20:16:47 (4500): No heartbeat from core client for 30 sec - exiting 20:16:48 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=424, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:09:22 (4744): No heartbeat from core client for 30 sec - exiting 08:09:23 (4744): No heartbeat from core client for 30 sec - exiting 08:09:25 (4744): No heartbeat from core client for 30 sec - exiting 08:09:26 (4744): No heartbeat from core client for 30 sec - exiting 08:09:27 (4744): No heartbeat from core client for 30 sec - exiting 08:09:28 (4744): No heartbeat from core client for 30 sec - exiting 08:09:29 (4744): No heartbeat from core client for 30 sec - exiting 08:09:30 (4744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:31 (4744): No heartbeat from core client for 30 sec - exiting 08:09:32 (4744): No heartbeat from core client for 30 sec - exiting 08:09:33 (4744): No heartbeat from core client for 30 sec - exiting 08:09:34 (4744): No heartbeat from core client for 30 sec - exiting 08:09:36 (4744): No heartbeat from core client for 30 sec - exiting 08:09:37 (4744): No heartbeat from core client for 30 sec - exiting 08:09:38 (4744): No heartbeat from core client for 30 sec - exiting 08:09:39 (4744): No heartbeat from core client for 30 sec - exiting 08:09:40 (4744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3692, iMonCtr=1 Model crash detected, will try to restart... 19:44:23 (4136): No heartbeat from core client for 30 sec - exiting 19:44:24 (4136): No heartbeat from core client for 30 sec - exiting 19:44:25 (4136): No heartbeat from core client for 30 sec - exiting 19:44:27 (4136): No heartbeat from core client for 30 sec - exiting 19:44:28 (4136): No heartbeat from core client for 30 sec - exiting 19:44:29 (4136): No heartbeat from core client for 30 sec - exiting 19:44:30 (4136): No heartbeat from core client for 30 sec - exiting 19:44:31 (4136): No heartbeat from core client for 30 sec - exiting 19:44:32 (4136): No heartbeat from core client for 30 sec - exiting 19:44:33 (4136): No heartbeat from core client for 30 sec - exiting 19:44:34 (4136): No heartbeat from core client for 30 sec - exiting 19:44:35 (4136): No heartbeat from core client for 30 sec - exiting 19:44:36 (4136): No heartbeat from core client for 30 sec - exiting 19:44:38 (4136): No heartbeat from core client for 30 sec - exiting 19:44:39 (4136): No heartbeat from core client for 30 sec - exiting 19:44:40 (4136): No heartbeat from core client for 30 sec - exiting 19:44:41 (4136): No heartbeat from core client for 30 sec - exiting 19:44:42 (4136): No heartbeat from core client for 30 sec - exiting 19:44:43 (4136): No heartbeat from core client for 30 sec - exiting 19:44:44 (4136): No heartbeat from core client for 30 sec - exiting 19:44:45 (4136): No heartbeat from core client for 30 sec - exiting 19:44:47 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:26 (4048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CCPDN Monitor - Quit request from BOINC... 20:09:25 (3656): No heartbeat from core client for 30 sec - exiting 20:09:27 (3656): No heartbeat from core client for 30 sec - exiting 20:09:28 (3656): No heartbeat from core client for 30 sec - exiting 20:09:29 (3656): No heartbeat from core client for 30 sec - exiting 20:09:30 (3656): No heartbeat from core client for 30 sec - exiting 20:09:31 (3656): No heartbeat from core client for 30 sec - exiting 20:09:32 (3656): No heartbeat from core client for 30 sec - exiting 20:09:34 (3656): No heartbeat from core client for 30 sec - exiting 20:09:35 (3656): No heartbeat from core client for 30 sec - exiting 20:09:36 (3656): No heartbeat from core client for 30 sec - exiting 20:09:37 (3656): No heartbeat from core client for 30 sec - exiting 20:09:38 (3656): No heartbeat from core client for 30 sec - exiting 20:09:39 (3656): No heartbeat from core client for 30 sec - exiting 20:09:40 (3656): No heartbeat from core client for 30 sec - exiting 20:09:41 (3656): No heartbeat from core client for 30 sec - exiting 20:09:42 (3656): No heartbeat from core client for 30 sec - exiting 20:09:43 (3656): No heartbeat from core client for 30 sec - exiting 20:09:45 (3656): No heartbeat from core client for 30 sec - exiting 20:09:46 (3656): No heartbeat from core client for 30 sec - exiting 20:09:47 (3656): No heartbeat from core client for 30 sec - exiting 20:09:48 (3656): No heartbeat from core client for 30 sec - exiting 20:09:49 (3656): No heartbeat from core client for 30 sec - exiting 20:09:50 (3656): No heartbeat from core client for 30 sec - exiting 20:09:51 (3656): No heartbeat from core client for 30 sec - exiting 20:09:52 (3656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:09:53 (3656): No heartbeat from core client for 30 sec - exiting 20:09:54 (3656): No heartbeat from core client for 30 sec - exiting 20:09:56 (3656): No heartbeat from core client for 30 sec - exiting 20:09:57 (3656): No heartbeat from core client for 30 sec - exiting 20:09:58 (3656): No heartbeat from core client for 30 sec - exiting 20:09:59 (3656): No heartbeat from core client for 30 sec - exiting 20:10:00 (3656): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4352, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:55:33 (2864): No heartbeat from core client for 30 sec - exiting 08:55:34 (2864): No heartbeat from core client for 30 sec - exiting 08:55:35 (2864): No heartbeat from core client for 30 sec - exiting 08:55:36 (2864): No heartbeat from core client for 30 sec - exiting 08:55:37 (2864): No heartbeat from core client for 30 sec - exiting 08:55:38 (2864): No heartbeat from core client for 30 sec - exiting 08:55:39 (2864): No heartbeat from core client for 30 sec - exiting 08:55:40 (2864): No heartbeat from core client for 30 sec - exiting 08:55:41 (2864): No heartbeat from core client for 30 sec - exiting 08:55:43 (2864): No heartbeat from core client for 30 sec - exiting 08:55:44 (2864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:45 (2864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:33:53 (4928): No heartbeat from core client for 30 sec - exiting 10:33:54 (4928): No heartbeat from core client for 30 sec - exiting 10:33:55 (4928): No heartbeat from core client for 30 sec - exiting 10:33:57 (4928): No heartbeat from core client for 30 sec - exiting 10:33:58 (4928): No heartbeat from core client for 30 sec - exiting 10:33:59 (4928): No heartbeat from core client for 30 sec - exiting 10:34:00 (4928): No heartbeat from core client for 30 sec - exiting 10:34:01 (4928): No heartbeat from core client for 30 sec - exiting 10:34:02 (4928): No heartbeat from core client for 30 sec - exiting 10:34:03 (4928): No heartbeat from core client for 30 sec - exiting 10:34:04 (4928): No heartbeat from core client for 30 sec - exiting 10:34:05 (4928): No heartbeat from core client for 30 sec - exiting 10:34:07 (4928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2680, iMonCtr=1 Model crash detected, will try to restart... 13:21:37 (4188): No heartbeat from core client for 30 sec - exiting 13:21:38 (4188): No heartbeat from core client for 30 sec - exiting 13:21:39 (4188): No heartbeat from core client for 30 sec - exiting 13:21:40 (4188): No heartbeat from core client for 30 sec - exiting 13:21:41 (4188): No heartbeat from core client for 30 sec - exiting 13:21:42 (4188): No heartbeat from core client for 30 sec - exiting 13:21:43 (4188): No heartbeat from core client for 30 sec - exiting 13:21:44 (4188): No heartbeat from core client for 30 sec - exiting 13:21:46 (4188): No heartbeat from core client for 30 sec - exiting 13:21:47 (4188): No heartbeat from core client for 30 sec - exiting 13:21:48 (4188): No heartbeat from core client for 30 sec - exiting 13:21:49 (4188): No heartbeat from core client for 30 sec - exiting 13:21:50 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:10:48 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4104, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:30 (5808): No heartbeat from core client for 30 sec - exiting 08:16:31 (5808): No heartbeat from core client for 30 sec - exiting 08:16:32 (5808): No heartbeat from core client for 30 sec - exiting 08:16:33 (5808): No heartbeat from core client for 30 sec - exiting 08:16:34 (5808): No heartbeat from core client for 30 sec - exiting 08:16:35 (5808): No heartbeat from core client for 30 sec - exiting 08:16:36 (5808): No heartbeat from core client for 30 sec - exiting 08:16:38 (5808): No heartbeat from core client for 30 sec - exiting 08:16:39 (5808): No heartbeat from core client for 30 sec - exiting 08:16:40 (5808): No heartbeat from core client for 30 sec - exiting 08:16:41 (5808): No heartbeat from core client for 30 sec - exiting 08:16:42 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:40:26 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2011 13:44:32	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	259,200	577,819	2.2292
31 Oct 2011 13:44:32	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	233,280	521,158	2.2340
18 Oct 2011 09:02:04	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	207,360	466,741	2.2509
14 Oct 2011 21:01:41	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	181,440	409,423	2.2565
12 Oct 2011 09:10:36	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	155,520	351,165	2.2580
09 Oct 2011 09:17:15	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	129,600	294,294	2.2708
06 Oct 2011 12:17:34	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	103,680	235,455	2.2710
03 Oct 2011 19:48:36	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	77,760	176,967	2.2758
29 Sep 2011 16:33:13	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	51,840	116,447	2.2463
26 Sep 2011 19:42:24	1149830	13415452	hadcm3n_t4b7_1940_40_007443762_4	25,920	57,577	2.2213