Task 12830735

Name	hadcm3n_p618_1900_40_007225404_0
Workunit	7423644
Created	26 Apr 2011, 15:34:46 UTC
Sent	27 Apr 2011, 19:25:01 UTC
Report deadline	28 Jul 2011, 2:52:12 UTC
Received	14 May 2011, 17:06:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1072815
Run time	5 days 13 hours 51 min 47 sec
CPU time	5 days 11 hours 48 min 18 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 18:47:52 (6412): No heartbeat from core client for 30 sec - exiting 18:47:53 (6412): No heartbeat from core client for 30 sec - exiting 18:47:54 (6412): No heartbeat from core client for 30 sec - exiting 18:47:55 (6412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:56 (6412): No heartbeat from core client for 30 sec - exiting 18:47:57 (6412): No heartbeat from core client for 30 sec - exiting 18:47:58 (6412): No heartbeat from core client for 30 sec - exiting 18:47:59 (6412): No heartbeat from core client for 30 sec - exiting 18:48:00 (6412): No heartbeat from core client for 30 sec - exiting 18:48:01 (6412): No heartbeat from core client for 30 sec - exiting 19:12:44 (5700): No heartbeat from core client for 30 sec - exiting 19:12:45 (5700): No heartbeat from core client for 30 sec - exiting 19:12:46 (5700): No heartbeat from core client for 30 sec - exiting 19:12:47 (5700): No heartbeat from core client for 30 sec - exiting 19:12:48 (5700): No heartbeat from core client for 30 sec - exiting 19:12:49 (5700): No heartbeat from core client for 30 sec - exiting 19:12:50 (5700): No heartbeat from core client for 30 sec - exiting 19:12:51 (5700): No heartbeat from core client for 30 sec - exiting 19:12:52 (5700): No heartbeat from core client for 30 sec - exiting 19:12:53 (5700): No heartbeat from core client for 30 sec - exiting 19:12:54 (5700): No heartbeat from core client for 30 sec - exiting 19:12:55 (5700): No heartbeat from core client for 30 sec - exiting 19:12:56 (5700): No heartbeat from core client for 30 sec - exiting 19:12:57 (5700): No heartbeat from core client for 30 sec - exiting 19:12:58 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:38:45 (3672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 11:27:08 (6088): No heartbeat from core client for 30 sec - exiting 11:27:09 (6088): No heartbeat from core client for 30 sec - exiting 11:27:10 (6088): No heartbeat from core client for 30 sec - exiting 11:27:11 (6088): No heartbeat from core client for 30 sec - exiting 11:27:12 (6088): No heartbeat from core client for 30 sec - exiting 11:27:13 (6088): No heartbeat from core client for 30 sec - exiting 11:27:14 (6088): No heartbeat from core client for 30 sec - exiting 11:27:15 (6088): No heartbeat from core client for 30 sec - exiting 11:27:16 (6088): No heartbeat from core client for 30 sec - exiting 11:27:17 (6088): No heartbeat from core client for 30 sec - exiting 11:27:18 (6088): No heartbeat from core client for 30 sec - exiting 11:27:19 (6088): No heartbeat from core client for 30 sec - exiting 11:27:20 (6088): No heartbeat from core client for 30 sec - exiting 11:27:21 (6088): No heartbeat from core client for 30 sec - exiting 11:27:22 (6088): No heartbeat from core client for 30 sec - exiting 11:27:23 (6088): No heartbeat from core client for 30 sec - exiting 11:27:24 (6088): No heartbeat from core client for 30 sec - exiting 11:27:25 (6088): No heartbeat from core client for 30 sec - exiting 11:27:26 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:57:49 (144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... 18:51:33 (5804): No heartbeat from core client for 30 sec - exiting 18:51:34 (5804): No heartbeat from core client for 30 sec - exiting 18:51:35 (5804): No heartbeat from core client for 30 sec - exiting 18:51:36 (5804): No heartbeat from core client for 30 sec - exiting 18:51:37 (5804): No heartbeat from core client for 30 sec - exiting 18:51:38 (5804): No heartbeat from core client for 30 sec - exiting 18:51:39 (5804): No heartbeat from core client for 30 sec - exiting 18:51:40 (5804): No heartbeat from core client for 30 sec - exiting 18:51:41 (5804): No heartbeat from core client for 30 sec - exiting 18:51:43 (5804): No heartbeat from core client for 30 sec - exiting 18:51:44 (5804): No heartbeat from core client for 30 sec - exiting 18:51:45 (5804): No heartbeat from core client for 30 sec - exiting 18:51:46 (5804): No heartbeat from core client for 30 sec - exiting 18:51:47 (5804): No heartbeat from core client for 30 sec - exiting 18:51:48 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:33 (5688): No heartbeat from core client for 30 sec - exiting 20:01:34 (5688): No heartbeat from core client for 30 sec - exiting 20:01:35 (5688): No heartbeat from core client for 30 sec - exiting 20:01:36 (5688): No heartbeat from core client for 30 sec - exiting 20:01:37 (5688): No heartbeat from core client for 30 sec - exiting 20:01:38 (5688): No heartbeat from core client for 30 sec - exiting 20:01:39 (5688): No heartbeat from core client for 30 sec - exiting 20:01:40 (5688): No heartbeat from core client for 30 sec - exiting 20:01:41 (5688): No heartbeat from core client for 30 sec - exiting 20:01:42 (5688): No heartbeat from core client for 30 sec - exiting 20:01:43 (5688): No heartbeat from core client for 30 sec - exiting 20:01:44 (5688): No heartbeat from core client for 30 sec - exiting 20:01:45 (5688): No heartbeat from core client for 30 sec - exiting 20:01:46 (5688): No heartbeat from core client for 30 sec - exiting 20:01:47 (5688): No heartbeat from core client for 30 sec - exiting 20:01:48 (5688): No heartbeat from core client for 30 sec - exiting 20:01:49 (5688): No heartbeat from core client for 30 sec - exiting 20:01:50 (5688): No heartbeat from core client for 30 sec - exiting 20:01:51 (5688): No heartbeat from core client for 30 sec - exiting 20:01:52 (5688): No heartbeat from core client for 30 sec - exiting 20:01:53 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:10 (3108): No heartbeat from core client for 30 sec - exiting 19:25:11 (3108): No heartbeat from core client for 30 sec - exiting 19:25:12 (3108): No heartbeat from core client for 30 sec - exiting 19:25:13 (3108): No heartbeat from core client for 30 sec - exiting 19:25:15 (3108): No heartbeat from core client for 30 sec - exiting 19:25:16 (3108): No heartbeat from core client for 30 sec - exiting 19:25:17 (3108): No heartbeat from core client for 30 sec - exiting 19:25:18 (3108): No heartbeat from core client for 30 sec - exiting 19:25:19 (3108): No heartbeat from core client for 30 sec - exiting 19:25:20 (3108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6432, iMonCtr=1 Model crash detected, will try to restart... 19:26:46 (5996): No heartbeat from core client for 30 sec - exiting 19:26:47 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:20 (5512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6556, iMonCtr=1 Model crash detected, will try to restart... 20:06:30 (5148): No heartbeat from core client for 30 sec - exiting 20:06:31 (5148): No heartbeat from core client for 30 sec - exiting 20:06:32 (5148): No heartbeat from core client for 30 sec - exiting 20:06:33 (5148): No heartbeat from core client for 30 sec - exiting 20:06:34 (5148): No heartbeat from core client for 30 sec - exiting 20:06:35 (5148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:06:36 (5148): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6496, iMonCtr=1 Model crash detected, will try to restart... 11:55:59 (5600): No heartbeat from core client for 30 sec - exiting 11:56:01 (5600): No heartbeat from core client for 30 sec - exiting 11:56:02 (5600): No heartbeat from core client for 30 sec - exiting 11:56:03 (5600): No heartbeat from core client for 30 sec - exiting 11:56:04 (5600): No heartbeat from core client for 30 sec - exiting 11:56:05 (5600): No heartbeat from core client for 30 sec - exiting 11:56:06 (5600): No heartbeat from core client for 30 sec - exiting 11:56:07 (5600): No heartbeat from core client for 30 sec - exiting 11:56:08 (5600): No heartbeat from core client for 30 sec - exiting 11:56:09 (5600): No heartbeat from core client for 30 sec - exiting 11:56:10 (5600): No heartbeat from core client for 30 sec - exiting 11:56:11 (5600): No heartbeat from core client for 30 sec - exiting 11:56:12 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:58:59 (3004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1692, iMonCtr=1 Model crash detected, will try to restart... 11:54:15 (5848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:13:16 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:24 (6756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:27 (6756): No heartbeat from core client for 30 sec - exiting 14:29:28 (6756): No heartbeat from core client for 30 sec - exiting 14:29:29 (6756): No heartbeat from core client for 30 sec - exiting 14:29:30 (6756): No heartbeat from core client for 30 sec - exiting 14:30:54 (5716): No heartbeat from core client for 30 sec - exiting 14:30:55 (5716): No heartbeat from core client for 30 sec - exiting 14:30:56 (5716): No heartbeat from core client for 30 sec - exiting 14:30:57 (5716): No heartbeat from core client for 30 sec - exiting 14:30:58 (5716): No heartbeat from core client for 30 sec - exiting 14:30:59 (5716): No heartbeat from core client for 30 sec - exiting 14:31:00 (5716): No heartbeat from core client for 30 sec - exiting 14:31:01 (5716): No heartbeat from core client for 30 sec - exiting 14:31:02 (5716): No heartbeat from core client for 30 sec - exiting 14:31:03 (5716): No heartbeat from core client for 30 sec - exiting 14:31:04 (5716): No heartbeat from core client for 30 sec - exiting 14:31:05 (5716): No heartbeat from core client for 30 sec - exiting 14:31:06 (5716): No heartbeat from core client for 30 sec - exiting 14:31:07 (5716): No heartbeat from core client for 30 sec - exiting 14:31:08 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:53:28 (6764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:53:29 (6764): No heartbeat from core client for 30 sec - exiting 19:53:30 (6764): No heartbeat from core client for 30 sec - exiting 19:54:17 (3548): No heartbeat from core client for 30 sec - exiting 19:54:18 (3548): No heartbeat from core client for 30 sec - exiting 19:54:19 (3548): No heartbeat from core client for 30 sec - exiting 19:54:20 (3548): No heartbeat from core client for 30 sec - exiting 19:54:21 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3752, iMonCtr=1 Model crash detected, will try to restart... 20:45:59 (5384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:49:49 (1376): No heartbeat from core client for 30 sec - exiting 20:49:50 (1376): No heartbeat from core client for 30 sec - exiting 20:49:51 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6612, iMonCtr=1 Model crash detected, will try to restart... 19:43:57 (5908): No heartbeat from core client for 30 sec - exiting 19:43:58 (5908): No heartbeat from core client for 30 sec - exiting 19:43:59 (5908): No heartbeat from core client for 30 sec - exiting 19:44:00 (5908): No heartbeat from core client for 30 sec - exiting 19:44:01 (5908): No heartbeat from core client for 30 sec - exiting 19:44:02 (5908): No heartbeat from core client for 30 sec - exiting 19:44:03 (5908): No heartbeat from core client for 30 sec - exiting 19:44:05 (5908): No heartbeat from core client for 30 sec - exiting 19:44:06 (5908): No heartbeat from core client for 30 sec - exiting 19:44:07 (5908): No heartbeat from core client for 30 sec - exiting 19:44:08 (5908): No heartbeat from core client for 30 sec - exiting 19:44:09 (5908): No heartbeat from core client for 30 sec - exiting 19:44:10 (5908): No heartbeat from core client for 30 sec - exiting 19:44:11 (5908): No heartbeat from core client for 30 sec - exiting 19:44:12 (5908): No heartbeat from core client for 30 sec - exiting 19:44:13 (5908): No heartbeat from core client for 30 sec - exiting 19:44:14 (5908): No heartbeat from core client for 30 sec - exiting 19:44:16 (5908): No heartbeat from core client for 30 sec - exiting 19:44:17 (5908): No heartbeat from core client for 30 sec - exiting 19:44:18 (5908): No heartbeat from core client for 30 sec - exiting 19:44:19 (5908): No heartbeat from core client for 30 sec - exiting 19:44:20 (5908): No heartbeat from core client for 30 sec - exiting 19:44:21 (5908): No heartbeat from core client for 30 sec - exiting 19:44:22 (5908): No heartbeat from core client for 30 sec - exiting 19:44:23 (5908): No heartbeat from core client for 30 sec - exiting 19:44:24 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:21 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:57:26 (7100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6856, iMonCtr=1 Model crash detected, will try to restart... 19:20:20 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:33 (6020): No heartbeat from core client for 30 sec - exiting 19:42:34 (6020): No heartbeat from core client for 30 sec - exiting 19:42:36 (6020): No heartbeat from core client for 30 sec - exiting 19:42:37 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:38 (6020): No heartbeat from core client for 30 sec - exiting 19:42:39 (6020): No heartbeat from core client for 30 sec - exiting 19:44:24 (6692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:47:31 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:50 (5888): No heartbeat from core client for 30 sec - exiting 19:12:51 (5888): No heartbeat from core client for 30 sec - exiting 19:12:52 (5888): No heartbeat from core client for 30 sec - exiting 19:12:53 (5888): No heartbeat from core client for 30 sec - exiting 19:12:54 (5888): No heartbeat from core client for 30 sec - exiting 19:12:55 (5888): No heartbeat from core client for 30 sec - exiting 19:12:56 (5888): No heartbeat from core client for 30 sec - exiting 19:12:57 (5888): No heartbeat from core client for 30 sec - exiting 19:12:59 (5888): No heartbeat from core client for 30 sec - exiting 19:13:00 (5888): No heartbeat from core client for 30 sec - exiting 19:13:01 (5888): No heartbeat from core client for 30 sec - exiting 19:13:02 (5888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3632, iMonCtr=1 Model crash detected, will try to restart... 11:33:59 (4532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:18 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:22:41 (5576): No heartbeat from core client for 30 sec - exiting 12:22:49 (5576): No heartbeat from core client for 30 sec - exiting 12:22:50 (5576): No heartbeat from core client for 30 sec - exiting 12:22:52 (5576): No heartbeat from core client for 30 sec - exiting 12:22:53 (5576): No heartbeat from core client for 30 sec - exiting 12:22:54 (5576): No heartbeat from core client for 30 sec - exiting 12:22:55 (5576): No heartbeat from core client for 30 sec - exiting 12:22:56 (5576): No heartbeat from core client for 30 sec - exiting 12:22:57 (5576): No heartbeat from core client for 30 sec - exiting 12:22:58 (5576): No heartbeat from core client for 30 sec - exiting 12:23:00 (5576): No heartbeat from core client for 30 sec - exiting 12:23:01 (5576): No heartbeat from core client for 30 sec - exiting 12:25:00 (5244): No heartbeat from core client for 30 sec - exiting 12:25:01 (5244): No heartbeat from core client for 30 sec - exiting 12:25:02 (5244): No heartbeat from core client for 30 sec - exiting 12:25:03 (5244): No heartbeat from core client for 30 sec - exiting 12:25:04 (5244): No heartbeat from core client for 30 sec - exiting 12:25:05 (5244): No heartbeat from core client for 30 sec - exiting 12:25:06 (5244): No heartbeat from core client for 30 sec - exiting 12:25:07 (5244): No heartbeat from core client for 30 sec - exiting 12:25:08 (5244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:09 (5244): No heartbeat from core client for 30 sec - exiting 17:31:34 (2464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 May 2011 17:22:01	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	259,200	474,483	1.8306
12 May 2011 22:55:55	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	233,280	426,330	1.8275
10 May 2011 20:55:51	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	207,360	378,737	1.8265
08 May 2011 19:00:02	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	181,440	330,725	1.8228
07 May 2011 18:11:46	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	155,520	282,389	1.8158
05 May 2011 23:21:17	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	129,600	234,778	1.8116
03 May 2011 22:24:38	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	103,680	188,571	1.8188
01 May 2011 20:32:26	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	77,760	141,383	1.8182
30 Apr 2011 20:16:46	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	51,840	94,058	1.8144
29 Apr 2011 21:03:00	1072815	12830735	hadcm3n_p618_1900_40_007225404_0	25,920	46,785	1.8050