Task 15673397

Name	hadcm3n_3exq_1940_40_008268280_4
Workunit	8423404
Created	19 Mar 2013, 22:33:53 UTC
Sent	19 Mar 2013, 22:33:56 UTC
Report deadline	19 Jun 2013, 6:01:07 UTC
Received	18 Jan 2014, 22:07:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1185891
Run time	22 days 17 hours 41 min 36 sec
CPU time	22 days 11 hours 39 min 20 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	0.84 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... 22:11:36 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:21 (2352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:05 (2804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:35:35 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:46 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:59 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=940, iMonCtr=1 Model crash detected, will try to restart... 21:02:20 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:02:37 (308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5556, iMonCtr=1 Model crash detected, will try to restart... 20:52:49 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:33:16 (420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:56 (1944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:44 (932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5152, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:15 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:58:37 (6336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:49:47 (6124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:21 (3032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:34:45 (5056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:03:53 (3924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:54:48 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:28 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:25:33 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=928, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:54:37 (5412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:54:38 (5412): No heartbeat from core client for 30 sec - exiting 20:26:04 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:14:24 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1376, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7240, iMonCtr=1 Model crash detected, will try to restart... 21:26:51 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:31:13 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:37:12 (2904): No heartbeat from core client for 30 sec - exiting 21:37:13 (2904): No heartbeat from core client for 30 sec - exiting 21:37:14 (2904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:37:15 (2904): No heartbeat from core client for 30 sec - exiting 21:37:16 (2904): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:46:00 (5624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6188, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5876, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2648, iMonCtr=1 Model crash detected, will try to restart... 20:26:55 (5200): No heartbeat from core client for 30 sec - exiting 20:26:56 (5200): No heartbeat from core client for 30 sec - exiting 20:26:58 (5200): No heartbeat from core client for 30 sec - exiting 20:26:59 (5200): No heartbeat from core client for 30 sec - exiting 20:27:00 (5200): No heartbeat from core client for 30 sec - exiting 20:27:01 (5200): No heartbeat from core client for 30 sec - exiting 20:27:02 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=1 Model crash detected, will try to restart... 20:40:35 (5828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:40:36 (5828): No heartbeat from core client for 30 sec - exiting 20:23:23 (5584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:32:42 (6048): No heartbeat from core client for 30 sec - exiting 19:32:44 (6048): No heartbeat from core client for 30 sec - exiting 19:32:45 (6048): No heartbeat from core client for 30 sec - exiting 19:32:46 (6048): No heartbeat from core client for 30 sec - exiting 19:32:47 (6048): No heartbeat from core client for 30 sec - exiting 19:32:48 (6048): No heartbeat from core client for 30 sec - exiting 19:32:49 (6048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:42:13 (6188): No heartbeat from core client for 30 sec - exiting 22:42:14 (6188): No heartbeat from core client for 30 sec - exiting 22:42:15 (6188): No heartbeat from core client for 30 sec - exiting 22:42:16 (6188): No heartbeat from core client for 30 sec - exiting 22:42:17 (6188): No heartbeat from core client for 30 sec - exiting 22:42:18 (6188): No heartbeat from core client for 30 sec - exiting 22:42:19 (6188): No heartbeat from core client for 30 sec - exiting 22:42:20 (6188): No heartbeat from core client for 30 sec - exiting 22:42:21 (6188): No heartbeat from core client for 30 sec - exiting 22:42:22 (6188): No heartbeat from core client for 30 sec - exiting 22:42:23 (6188): No heartbeat from core client for 30 sec - exiting 22:42:24 (6188): No heartbeat from core client for 30 sec - exiting 22:42:25 (6188): No heartbeat from core client for 30 sec - exiting 22:42:26 (6188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:38:25 (6244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:59:42 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:29 (5716): No heartbeat from core client for 30 sec - exiting 19:33:30 (5716): No heartbeat from core client for 30 sec - exiting 19:33:31 (5716): No heartbeat from core client for 30 sec - exiting 19:33:33 (5716): No heartbeat from core client for 30 sec - exiting 19:33:34 (5716): No heartbeat from core client for 30 sec - exiting 19:33:35 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:22:28 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:36:17 (5424): No heartbeat from core client for 30 sec - exiting 20:36:18 (5424): No heartbeat from core client for 30 sec - exiting 20:36:19 (5424): No heartbeat from core client for 30 sec - exiting 20:36:20 (5424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:19:20 (7152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... 19:26:08 (1440): No heartbeat from core client for 30 sec - exiting 19:26:09 (1440): No heartbeat from core client for 30 sec - exiting 19:26:10 (1440): No heartbeat from core client for 30 sec - exiting 19:26:11 (1440): No heartbeat from core client for 30 sec - exiting 19:26:12 (1440): No heartbeat from core client for 30 sec - exiting 19:26:14 (1440): No heartbeat from core client for 30 sec - exiting 19:26:15 (1440): No heartbeat from core client for 30 sec - exiting 19:26:16 (1440): No heartbeat from core client for 30 sec - exiting 19:26:17 (1440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6020, iMonCtr=1 Model crash detected, will try to restart... 18:36:37 (720): No heartbeat from core client for 30 sec - exiting 18:36:38 (720): No heartbeat from core client for 30 sec - exiting 18:36:40 (720): No heartbeat from core client for 30 sec - exiting 18:36:41 (720): No heartbeat from core client for 30 sec - exiting 18:36:42 (720): No heartbeat from core client for 30 sec - exiting 18:36:43 (720): No heartbeat from core client for 30 sec - exiting 18:36:44 (720): No heartbeat from core client for 30 sec - exiting 18:36:45 (720): No heartbeat from core client for 30 sec - exiting 18:36:46 (720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:07 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:15:26 (4060): No heartbeat from core client for 30 sec - exiting 14:15:27 (4060): No heartbeat from core client for 30 sec - exiting 14:15:28 (4060): No heartbeat from core client for 30 sec - exiting 14:15:29 (4060): No heartbeat from core client for 30 sec - exiting 14:15:30 (4060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7064, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:55:14 (5704): No heartbeat from core client for 30 sec - exiting 19:55:15 (5704): No heartbeat from core client for 30 sec - exiting 19:55:16 (5704): No heartbeat from core client for 30 sec - exiting 19:55:17 (5704): No heartbeat from core client for 30 sec - exiting 19:55:18 (5704): No heartbeat from core client for 30 sec - exiting 19:55:19 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:30:49 (5936): No heartbeat from core client for 30 sec - exiting 19:30:50 (5936): No heartbeat from core client for 30 sec - exiting 19:30:51 (5936): No heartbeat from core client for 30 sec - exiting 19:30:52 (5936): No heartbeat from core client for 30 sec - exiting 19:30:53 (5936): No heartbeat from core client for 30 sec - exiting 19:30:54 (5936): No heartbeat from core client for 30 sec - exiting 19:30:55 (5936): No heartbeat from core client for 30 sec - exiting 19:30:56 (5936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:43:12 (5648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jan 2014 22:07:41	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	259,200	1,942,752	7.4952
06 Jan 2014 21:49:50	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	233,280	1,837,003	7.8747
10 Dec 2013 21:04:22	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	207,360	1,724,972	8.3187
18 Aug 2013 18:46:45	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	181,440	792,475	4.3677
23 Jul 2013 21:55:28	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	155,520	677,830	4.3585
18 Jun 2013 21:45:37	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	129,600	562,348	4.3391
02 Jun 2013 20:37:36	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	103,680	457,298	4.4107
19 May 2013 20:04:36	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	77,760	341,973	4.3978
28 Apr 2013 19:06:36	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	51,840	227,322	4.3851
08 Apr 2013 21:25:23	1185891	15673397	hadcm3n_3exq_1940_40_008268280_4	25,920	113,910	4.3947