Task 14616993

Name	hadcm3n_105r_1980_40_007954738_0
Workunit	8109850
Created	2 May 2012, 10:53:28 UTC
Sent	2 May 2012, 10:54:24 UTC
Report deadline	1 Aug 2012, 18:21:35 UTC
Received	18 Jul 2012, 22:51:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1119216
Run time	39 days 2 hours 2 min 40 sec
CPU time	34 days 13 hours 5 min 15 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.19 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 05:48:12 (4200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7324, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:01:48 (4492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:50 (4492): No heartbeat from core client for 30 sec - exiting 22:01:51 (4492): No heartbeat from core client for 30 sec - exiting 22:01:52 (4492): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3964, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5596, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6028, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... 07:42:43 (860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6488, iMonCtr=1 Model crash detected, will try to restart... 21:20:09 (6680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:11 (6680): No heartbeat from core client for 30 sec - exiting 21:20:12 (6680): No heartbeat from core client for 30 sec - exiting 21:20:13 (6680): No heartbeat from core client for 30 sec - exiting 21:20:14 (6680): No heartbeat from core client for 30 sec - exiting 21:20:15 (6680): No heartbeat from core client for 30 sec - exiting 21:33:30 (6380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:40:12 (3992): No heartbeat from core client for 30 sec - exiting 06:40:13 (3992): No heartbeat from core client for 30 sec - exiting 06:40:14 (3992): No heartbeat from core client for 30 sec - exiting 06:40:15 (3992): No heartbeat from core client for 30 sec - exiting 06:40:16 (3992): No heartbeat from core client for 30 sec - exiting 06:40:17 (3992): No heartbeat from core client for 30 sec - exiting 06:40:18 (3992): No heartbeat from core client for 30 sec - exiting 06:40:19 (3992): No heartbeat from core client for 30 sec - exiting 06:40:20 (3992): No heartbeat from core client for 30 sec - exiting 06:40:21 (3992): No heartbeat from core client for 30 sec - exiting 06:40:22 (3992): No heartbeat from core client for 30 sec - exiting 06:40:23 (3992): No heartbeat from core client for 30 sec - exiting 06:40:24 (3992): No heartbeat from core client for 30 sec - exiting 06:40:25 (3992): No heartbeat from core client for 30 sec - exiting 06:40:26 (3992): No heartbeat from core client for 30 sec - exiting 06:40:27 (3992): No heartbeat from core client for 30 sec - exiting 06:40:28 (3992): No heartbeat from core client for 30 sec - exiting 06:40:29 (3992): No heartbeat from core client for 30 sec - exiting 06:40:30 (3992): No heartbeat from core client for 30 sec - exiting 06:40:31 (3992): No heartbeat from core client for 30 sec - exiting 06:40:32 (3992): No heartbeat from core client for 30 sec - exiting 06:40:33 (3992): No heartbeat from core client for 30 sec - exiting 06:40:34 (3992): No heartbeat from core client for 30 sec - exiting 06:40:35 (3992): No heartbeat from core client for 30 sec - exiting 06:40:36 (3992): No heartbeat from core client for 30 sec - exiting 06:40:37 (3992): No heartbeat from core client for 30 sec - exiting 06:40:38 (3992): No heartbeat from core client for 30 sec - exiting 06:40:39 (3992): No heartbeat from core client for 30 sec - exiting 06:40:40 (3992): No heartbeat from core client for 30 sec - exiting 06:40:41 (3992): No heartbeat from core client for 30 sec - exiting 06:40:42 (3992): No heartbeat from core client for 30 sec - exiting 06:40:43 (3992): No heartbeat from core client for 30 sec - exiting 06:40:44 (3992): No heartbeat from core client for 30 sec - exiting 06:40:45 (3992): No heartbeat from core client for 30 sec - exiting 06:40:46 (3992): No heartbeat from core client for 30 sec - exiting 06:40:47 (3992): No heartbeat from core client for 30 sec - exiting 06:40:48 (3992): No heartbeat from core client for 30 sec - exiting 06:40:49 (3992): No heartbeat from core client for 30 sec - exiting 06:40:50 (3992): No heartbeat from core client for 30 sec - exiting 06:40:51 (3992): No heartbeat from core client for 30 sec - exiting 06:40:52 (3992): No heartbeat from core client for 30 sec - exiting 06:40:53 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:58:30 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:32 (7648): No heartbeat from core client for 30 sec - exiting 07:58:33 (7648): No heartbeat from core client for 30 sec - exiting 07:58:34 (7648): No heartbeat from core client for 30 sec - exiting 07:58:35 (7648): No heartbeat from core client for 30 sec - exiting 07:58:36 (7648): No heartbeat from core client for 30 sec - exiting 07:58:37 (7648): No heartbeat from core client for 30 sec - exiting 07:58:38 (7648): No heartbeat from core client for 30 sec - exiting 07:58:39 (7648): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:33:58 (6484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:13 (4328): No heartbeat from core client for 30 sec - exiting 09:57:14 (4328): No heartbeat from core client for 30 sec - exiting 09:57:15 (4328): No heartbeat from core client for 30 sec - exiting 09:57:16 (4328): No heartbeat from core client for 30 sec - exiting 09:57:17 (4328): No heartbeat from core client for 30 sec - exiting 09:57:19 (4328): No heartbeat from core client for 30 sec - exiting 09:57:20 (4328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:21 (4328): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 20:48:19 (3492): No heartbeat from core client for 30 sec - exiting 20:48:20 (3492): No heartbeat from core client for 30 sec - exiting 20:48:21 (3492): No heartbeat from core client for 30 sec - exiting 20:48:22 (3492): No heartbeat from core client for 30 sec - exiting 20:48:23 (3492): No heartbeat from core client for 30 sec - exiting 20:48:25 (3492): No heartbeat from core client for 30 sec - exiting 20:48:26 (3492): No heartbeat from core client for 30 sec - exiting 20:48:27 (3492): No heartbeat from core client for 30 sec - exiting 20:48:28 (3492): No heartbeat from core client for 30 sec - exiting 20:48:29 (3492): No heartbeat from core client for 30 sec - exiting 20:48:30 (3492): No heartbeat from core client for 30 sec - exiting 20:48:31 (3492): No heartbeat from core client for 30 sec - exiting 20:48:32 (3492): No heartbeat from core client for 30 sec - exiting 20:48:33 (3492): No heartbeat from core client for 30 sec - exiting 20:48:34 (3492): No heartbeat from core client for 30 sec - exiting 20:48:35 (3492): No heartbeat from core client for 30 sec - exiting 20:48:37 (3492): No heartbeat from core client for 30 sec - exiting 20:48:38 (3492): No heartbeat from core client for 30 sec - exiting 20:48:39 (3492): No heartbeat from core client for 30 sec - exiting 20:48:40 (3492): No heartbeat from core client for 30 sec - exiting 20:48:41 (3492): No heartbeat from core client for 30 sec - exiting 20:48:42 (3492): No heartbeat from core client for 30 sec - exiting 20:48:43 (3492): No heartbeat from core client for 30 sec - exiting 20:48:44 (3492): No heartbeat from core client for 30 sec - exiting 20:48:45 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:56:32 (3628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:56:33 (3628): No heartbeat from core client for 30 sec - exiting 08:56:34 (3628): No heartbeat from core client for 30 sec - exiting 08:56:35 (3628): No heartbeat from core client for 30 sec - exiting 08:56:36 (3628): No heartbeat from core client for 30 sec - exiting 08:56:38 (3628): No heartbeat from core client for 30 sec - exiting 08:56:39 (3628): No heartbeat from core client for 30 sec - exiting 08:56:40 (3628): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:58:37 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:12:49 (5132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:14 (7932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jul 2012 22:54:07	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	777,600	2,984,693	3.8383
17 Jul 2012 09:50:31	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	751,680	2,883,425	3.8360
16 Jul 2012 00:54:35	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	725,760	2,778,024	3.8277
14 Jul 2012 16:21:45	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	699,840	2,676,560	3.8245
13 Jul 2012 10:49:52	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	673,920	2,574,122	3.8196
11 Jul 2012 15:40:15	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	648,000	2,469,796	3.8114
10 Jul 2012 08:50:23	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	622,080	2,367,692	3.8061
09 Jul 2012 00:03:13	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	596,160	2,263,816	3.7973
07 Jul 2012 13:31:07	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	570,240	2,158,168	3.7847
06 Jul 2012 04:33:27	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	544,320	2,055,579	3.7764
04 Jul 2012 15:38:13	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	518,400	1,948,693	3.7591
02 Jul 2012 22:01:01	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	492,480	1,846,723	3.7498
02 Jul 2012 11:07:22	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	466,560	1,741,276	3.7322
26 Jun 2012 12:19:11	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	440,640	1,635,647	3.7120
24 Jun 2012 17:35:33	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	414,720	1,527,237	3.6826
23 Jun 2012 07:31:54	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	388,800	1,429,448	3.6766
19 Jun 2012 03:51:19	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	362,880	1,326,650	3.6559
17 Jun 2012 00:17:35	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	336,960	1,219,107	3.6180
15 Jun 2012 21:49:11	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	311,040	1,127,577	3.6252
14 Jun 2012 15:10:09	1119216	14616993	hadcm3n_105r_1980_40_007954738_0	285,120	1,032,971	3.6229