Task 16216976

Name	hadcm3n_7a1g_1980_40_008424295_1
Workunit	8575151
Created	13 Jan 2014, 22:45:35 UTC
Sent	13 Jan 2014, 22:47:02 UTC
Report deadline	15 Apr 2014, 6:14:13 UTC
Received	18 Mar 2014, 17:58:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1130821
Run time	61 days 10 hours 57 min 15 sec
CPU time	43 days 17 hours 2 min 16 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	0.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 13:31:04 (30524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:05 (30524): No heartbeat from core client for 30 sec - exiting 12:04:36 (38944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:50 (26084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:58:42 (11700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:58:43 (11700): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:33:03 (40384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:33:04 (40384): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 12:42:15 (37072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:16 (37072): No heartbeat from core client for 30 sec - exiting 12:58:16 (25328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:17 (25328): No heartbeat from core client for 30 sec - exiting 16:26:37 (40496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:39 (40496): No heartbeat from core client for 30 sec - exiting 16:26:40 (40496): No heartbeat from core client for 30 sec - exiting 16:26:41 (40496): No heartbeat from core client for 30 sec - exiting 16:26:42 (40496): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 13:21:17 (36716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:18 (36716): No heartbeat from core client for 30 sec - exiting 13:21:19 (36716): No heartbeat from core client for 30 sec - exiting 12:57:43 (41916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:44 (41916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:16:14 (37680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:34:39 (43908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50784, iMonCtr=1 Model crash detected, will try to restart... 14:01:17 (5276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:01:19 (5276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12584, iMonCtr=1 Model crash detected, will try to restart... 12:37:46 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:47 (5760): No heartbeat from core client for 30 sec - exiting 12:36:58 (9932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:59 (9932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:35:24 (15808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:46:30 (18760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:46:31 (18760): No heartbeat from core client for 30 sec - exiting 13:07:59 (20896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:00 (20896): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17620, iMonCtr=1 Model crash detected, will try to restart... 19:08:16 (5508): No heartbeat from core client for 30 sec - exiting 19:08:17 (5508): No heartbeat from core client for 30 sec - exiting 19:08:18 (5508): No heartbeat from core client for 30 sec - exiting 19:08:23 (5508): No heartbeat from core client for 30 sec - exiting 19:08:26 (5508): No heartbeat from core client for 30 sec - exiting 19:08:28 (5508): No heartbeat from core client for 30 sec - exiting 19:08:29 (5508): No heartbeat from core client for 30 sec - exiting 19:08:32 (5508): No heartbeat from core client for 30 sec - exiting 19:08:34 (5508): No heartbeat from core client for 30 sec - exiting 19:08:36 (5508): No heartbeat from core client for 30 sec - exiting 19:08:37 (5508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:39 (5508): No heartbeat from core client for 30 sec - exiting 14:01:05 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:03 (7500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:04 (7500): No heartbeat from core client for 30 sec - exiting 13:23:09 (12440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:55:36 (16096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:47:50 (4432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:43:25 (9296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:07:31 (2168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:51 (2324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:30 (18888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:51 (10620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:51 (9284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:52 (9284): No heartbeat from core client for 30 sec - exiting 03:22:20 (6912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:25:34 (19300): No heartbeat from core client for 30 sec - exiting 22:25:35 (19300): No heartbeat from core client for 30 sec - exiting 22:25:36 (19300): No heartbeat from core client for 30 sec - exiting 22:25:37 (19300): No heartbeat from core client for 30 sec - exiting 22:25:38 (19300): No heartbeat from core client for 30 sec - exiting 22:25:39 (19300): No heartbeat from core client for 30 sec - exiting 22:25:40 (19300): No heartbeat from core client for 30 sec - exiting 22:25:41 (19300): No heartbeat from core client for 30 sec - exiting 22:25:42 (19300): No heartbeat from core client for 30 sec - exiting 22:25:43 (19300): No heartbeat from core client for 30 sec - exiting 22:25:44 (19300): No heartbeat from core client for 30 sec - exiting 22:25:45 (19300): No heartbeat from core client for 30 sec - exiting 22:25:46 (19300): No heartbeat from core client for 30 sec - exiting 22:25:47 (19300): No heartbeat from core client for 30 sec - exiting 22:25:48 (19300): No heartbeat from core client for 30 sec - exiting 22:25:49 (19300): No heartbeat from core client for 30 sec - exiting 22:25:50 (19300): No heartbeat from core client for 30 sec - exiting 22:25:51 (19300): No heartbeat from core client for 30 sec - exiting 22:25:52 (19300): No heartbeat from core client for 30 sec - exiting 22:25:53 (19300): No heartbeat from core client for 30 sec - exiting 22:25:55 (19300): No heartbeat from core client for 30 sec - exiting 22:25:56 (19300): No heartbeat from core client for 30 sec - exiting 22:25:57 (19300): No heartbeat from core client for 30 sec - exiting 22:25:58 (19300): No heartbeat from core client for 30 sec - exiting 22:25:59 (19300): No heartbeat from core client for 30 sec - exiting 22:26:00 (19300): No heartbeat from core client for 30 sec - exiting 22:26:01 (19300): No heartbeat from core client for 30 sec - exiting 22:26:02 (19300): No heartbeat from core client for 30 sec - exiting 22:26:03 (19300): No heartbeat from core client for 30 sec - exiting 22:26:04 (19300): No heartbeat from core client for 30 sec - exiting 22:26:05 (19300): No heartbeat from core client for 30 sec - exiting 22:26:06 (19300): No heartbeat from core client for 30 sec - exiting 22:26:07 (19300): No heartbeat from core client for 30 sec - exiting 22:26:08 (19300): No heartbeat from core client for 30 sec - exiting 22:26:09 (19300): No heartbeat from core client for 30 sec - exiting 22:26:10 (19300): No heartbeat from core client for 30 sec - exiting 22:26:12 (19300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:34:21 (25188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:34:22 (25188): No heartbeat from core client for 30 sec - exiting 12:18:15 (23912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:18:16 (23912): No heartbeat from core client for 30 sec - exiting 12:30:59 (7456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:31:00 (7456): No heartbeat from core client for 30 sec - exiting 13:31:45 (30328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:46 (30328): No heartbeat from core client for 30 sec - exiting 13:39:38 (24288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:40 (24288): No heartbeat from core client for 30 sec - exiting 14:40:41 (18800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:42 (18800): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:37:43 (2356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:44 (2356): No heartbeat from core client for 30 sec - exiting 18:09:14 (5944): No heartbeat from core client for 30 sec - exiting 18:09:16 (5944): No heartbeat from core client for 30 sec - exiting 18:09:17 (5944): No heartbeat from core client for 30 sec - exiting 18:09:18 (5944): No heartbeat from core client for 30 sec - exiting 18:09:19 (5944): No heartbeat from core client for 30 sec - exiting 18:09:20 (5944): No heartbeat from core client for 30 sec - exiting 18:09:21 (5944): No heartbeat from core client for 30 sec - exiting 18:09:22 (5944): No heartbeat from core client for 30 sec - exiting 18:09:23 (5944): No heartbeat from core client for 30 sec - exiting 18:09:24 (5944): No heartbeat from core client for 30 sec - exiting 18:09:25 (5944): No heartbeat from core client for 30 sec - exiting 18:09:27 (5944): No heartbeat from core client for 30 sec - exiting 18:09:28 (5944): No heartbeat from core client for 30 sec - exiting 18:09:29 (5944): No heartbeat from core client for 30 sec - exiting 18:09:30 (5944): No heartbeat from core client for 30 sec - exiting 18:09:31 (5944): No heartbeat from core client for 30 sec - exiting 18:09:32 (5944): No heartbeat from core client for 30 sec - exiting 18:09:33 (5944): No heartbeat from core client for 30 sec - exiting 18:09:34 (5944): No heartbeat from core client for 30 sec - exiting 18:09:35 (5944): No heartbeat from core client for 30 sec - exiting 18:09:37 (5944): No heartbeat from core client for 30 sec - exiting 18:09:38 (5944): No heartbeat from core client for 30 sec - exiting 18:09:39 (5944): No heartbeat from core client for 30 sec - exiting 18:09:40 (5944): No heartbeat from core client for 30 sec - exiting 18:09:41 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:34:26 (5536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:27 (5536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:06:41 (9948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7996, iMonCtr=1 Model crash detected, will try to restart... 18:43:23 (1064): No heartbeat from core client for 30 sec - exiting 18:43:24 (1064): No heartbeat from core client for 30 sec - exiting 18:43:25 (1064): No heartbeat from core client for 30 sec - exiting 18:43:26 (1064): No heartbeat from core client for 30 sec - exiting 18:43:27 (1064): No heartbeat from core client for 30 sec - exiting 18:43:28 (1064): No heartbeat from core client for 30 sec - exiting 18:43:29 (1064): No heartbeat from core client for 30 sec - exiting 18:43:30 (1064): No heartbeat from core client for 30 sec - exiting 18:43:31 (1064): No heartbeat from core client for 30 sec - exiting 18:43:33 (1064): No heartbeat from core client for 30 sec - exiting 18:43:34 (1064): No heartbeat from core client for 30 sec - exiting 18:43:35 (1064): No heartbeat from core client for 30 sec - exiting 18:43:36 (1064): No heartbeat from core client for 30 sec - exiting 18:43:37 (1064): No heartbeat from core client for 30 sec - exiting 18:43:38 (1064): No heartbeat from core client for 30 sec - exiting 18:43:39 (1064): No heartbeat from core client for 30 sec - exiting 18:43:40 (1064): No heartbeat from core client for 30 sec - exiting 18:43:42 (1064): No heartbeat from core client for 30 sec - exiting 18:43:43 (1064): No heartbeat from core client for 30 sec - exiting 18:43:44 (1064): No heartbeat from core client for 30 sec - exiting 18:43:45 (1064): No heartbeat from core client for 30 sec - exiting 18:43:46 (1064): No heartbeat from core client for 30 sec - exiting 18:43:47 (1064): No heartbeat from core client for 30 sec - exiting 18:43:48 (1064): No heartbeat from core client for 30 sec - exiting 18:43:49 (1064): No heartbeat from core client for 30 sec - exiting 18:43:51 (1064): No heartbeat from core client for 30 sec - exiting 18:43:52 (1064): No heartbeat from core client for 30 sec - exiting 18:43:54 (1064): No heartbeat from core client for 30 sec - exiting 18:43:55 (1064): No heartbeat from core client for 30 sec - exiting 18:43:56 (1064): No heartbeat from core client for 30 sec - exiting 18:43:58 (1064): No heartbeat from core client for 30 sec - exiting 18:43:59 (1064): No heartbeat from core client for 30 sec - exiting 18:44:00 (1064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:01 (1064): No heartbeat from core client for 30 sec - exiting 18:44:03 (1064): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6008, iMonCtr=1 Model crash detected, will try to restart... 00:37:55 (5176): No heartbeat from core client for 30 sec - exiting 00:37:56 (5176): No heartbeat from core client for 30 sec - exiting 00:37:57 (5176): No heartbeat from core client for 30 sec - exiting 00:37:58 (5176): No heartbeat from core client for 30 sec - exiting 00:37:59 (5176): No heartbeat from core client for 30 sec - exiting 00:38:00 (5176): No heartbeat from core client for 30 sec - exiting 00:38:01 (5176): No heartbeat from core client for 30 sec - exiting 00:38:03 (5176): No heartbeat from core client for 30 sec - exiting 00:38:04 (5176): No heartbeat from core client for 30 sec - exiting 00:38:05 (5176): No heartbeat from core client for 30 sec - exiting 00:38:06 (5176): No heartbeat from core client for 30 sec - exiting 00:38:07 (5176): No heartbeat from core client for 30 sec - exiting 00:38:08 (5176): No heartbeat from core client for 30 sec - exiting 00:38:09 (5176): No heartbeat from core client for 30 sec - exiting 00:38:10 (5176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:16 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:18 (4996): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 13:29:54 (7956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:16 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:17 (5396): No heartbeat from core client for 30 sec - exiting 14:02:28 (10220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:02:30 (10220): No heartbeat from core client for 30 sec - exiting 12:24:09 (10360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B56E9F read attempt to address 0x408E8C5B Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15280, selfPID=15280, iMonCtr=1 Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Mar 2014 10:57:21	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	777,600	3,699,635	4.7578
14 Mar 2014 11:44:34	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	751,680	3,576,049	4.7574
12 Mar 2014 13:09:00	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	725,760	3,452,937	4.7577
10 Mar 2014 13:22:06	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	699,840	3,329,558	4.7576
08 Mar 2014 14:24:57	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	673,920	3,205,976	4.7572
06 Mar 2014 15:37:46	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	648,000	3,082,768	4.7574
03 Mar 2014 16:19:32	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	622,080	2,956,823	4.7531
28 Feb 2014 12:16:47	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	596,160	2,830,847	4.7485
26 Feb 2014 11:00:12	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	570,240	2,707,678	4.7483
24 Feb 2014 10:59:12	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	544,320	2,583,114	4.7456
22 Feb 2014 11:57:53	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	518,400	2,460,901	4.7471
20 Feb 2014 10:21:53	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	492,480	2,338,498	4.7484
18 Feb 2014 14:49:27	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	466,560	2,219,468	4.7571
16 Feb 2014 15:48:22	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	440,640	2,094,830	4.7541
14 Feb 2014 16:37:57	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	414,720	1,971,828	4.7546
12 Feb 2014 16:13:29	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	388,800	1,848,873	4.7553
10 Feb 2014 16:52:01	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	362,880	1,724,149	4.7513
08 Feb 2014 18:12:12	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	336,960	1,600,860	4.7509
06 Feb 2014 19:22:45	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	311,040	1,477,448	4.7500
04 Feb 2014 22:34:02	1130821	16216976	hadcm3n_7a1g_1980_40_008424295_1	285,120	1,356,557	4.7578