Task 15330024

Name	hadcm3n_zj6l_1880_40_008200984_2
Workunit	8356108
Created	4 Oct 2012, 17:29:58 UTC
Sent	4 Oct 2012, 17:30:23 UTC
Report deadline	4 Jan 2013, 0:57:34 UTC
Received	13 Oct 2012, 0:26:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1238808
Run time	4 days 0 hours 59 min 53 sec
CPU time	3 days 23 hours 56 min 40 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.01 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 09:18:22 (3420): No heartbeat from core client for 30 sec - exiting 09:18:24 (3420): No heartbeat from core client for 30 sec - exiting 09:18:25 (3420): No heartbeat from core client for 30 sec - exiting 09:18:26 (3420): No heartbeat from core client for 30 sec - exiting 09:18:27 (3420): No heartbeat from core client for 30 sec - exiting 09:18:28 (3420): No heartbeat from core client for 30 sec - exiting 09:18:29 (3420): No heartbeat from core client for 30 sec - exiting 09:18:30 (3420): No heartbeat from core client for 30 sec - exiting 09:18:31 (3420): No heartbeat from core client for 30 sec - exiting 09:18:32 (3420): No heartbeat from core client for 30 sec - exiting 09:18:33 (3420): No heartbeat from core client for 30 sec - exiting 09:18:35 (3420): No heartbeat from core client for 30 sec - exiting 09:18:36 (3420): No heartbeat from core client for 30 sec - exiting 09:18:37 (3420): No heartbeat from core client for 30 sec - exiting 09:18:38 (3420): No heartbeat from core client for 30 sec - exiting 09:18:39 (3420): No heartbeat from core client for 30 sec - exiting 09:18:40 (3420): No heartbeat from core client for 30 sec - exiting 09:18:41 (3420): No heartbeat from core client for 30 sec - exiting 09:18:42 (3420): No heartbeat from core client for 30 sec - exiting 09:18:43 (3420): No heartbeat from core client for 30 sec - exiting 09:18:44 (3420): No heartbeat from core client for 30 sec - exiting 09:18:45 (3420): No heartbeat from core client for 30 sec - exiting 09:18:47 (3420): No heartbeat from core client for 30 sec - exiting 09:18:48 (3420): No heartbeat from core client for 30 sec - exiting 09:18:49 (3420): No heartbeat from core client for 30 sec - exiting 09:18:50 (3420): No heartbeat from core client for 30 sec - exiting 09:18:51 (3420): No heartbeat from core client for 30 sec - exiting 09:18:52 (3420): No heartbeat from core client for 30 sec - exiting 09:18:53 (3420): No heartbeat from core client for 30 sec - exiting 09:18:54 (3420): No heartbeat from core client for 30 sec - exiting 09:18:55 (3420): No heartbeat from core client for 30 sec - exiting 09:18:56 (3420): No heartbeat from core client for 30 sec - exiting 09:18:57 (3420): No heartbeat from core client for 30 sec - exiting 09:18:58 (3420): No heartbeat from core client for 30 sec - exiting 09:18:59 (3420): No heartbeat from core client for 30 sec - exiting 09:19:00 (3420): No heartbeat from core client for 30 sec - exiting 09:19:01 (3420): No heartbeat from core client for 30 sec - exiting 09:19:02 (3420): No heartbeat from core client for 30 sec - exiting 09:19:03 (3420): No heartbeat from core client for 30 sec - exiting 09:19:04 (3420): No heartbeat from core client for 30 sec - exiting 09:19:05 (3420): No heartbeat from core client for 30 sec - exiting 09:19:06 (3420): No heartbeat from core client for 30 sec - exiting 09:19:07 (3420): No heartbeat from core client for 30 sec - exiting 09:19:08 (3420): No heartbeat from core client for 30 sec - exiting 09:19:09 (3420): No heartbeat from core client for 30 sec - exiting 09:19:10 (3420): No heartbeat from core client for 30 sec - exiting 09:19:11 (3420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6904, iMonCtr=1 Model crash detected, will try to restart... 09:16:34 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:12:54 (6344): No heartbeat from core client for 30 sec - exiting 21:12:55 (6344): No heartbeat from core client for 30 sec - exiting 21:12:56 (6344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6936, iMonCtr=1 Model crash detected, will try to restart... 09:19:17 (6088): No heartbeat from core client for 30 sec - exiting 09:19:18 (6088): No heartbeat from core client for 30 sec - exiting 09:19:20 (6088): No heartbeat from core client for 30 sec - exiting 09:19:21 (6088): No heartbeat from core client for 30 sec - exiting 09:19:22 (6088): No heartbeat from core client for 30 sec - exiting 09:19:23 (6088): No heartbeat from core client for 30 sec - exiting 09:19:24 (6088): No heartbeat from core client for 30 sec - exiting 09:19:25 (6088): No heartbeat from core client for 30 sec - exiting 09:19:26 (6088): No heartbeat from core client for 30 sec - exiting 09:19:27 (6088): No heartbeat from core client for 30 sec - exiting 09:19:28 (6088): No heartbeat from core client for 30 sec - exiting 09:19:29 (6088): No heartbeat from core client for 30 sec - exiting 09:19:30 (6088): No heartbeat from core client for 30 sec - exiting 09:19:32 (6088): No heartbeat from core client for 30 sec - exiting 09:19:33 (6088): No heartbeat from core client for 30 sec - exiting 09:19:34 (6088): No heartbeat from core client for 30 sec - exiting 09:19:35 (6088): No heartbeat from core client for 30 sec - exiting 09:19:36 (6088): No heartbeat from core client for 30 sec - exiting 09:19:37 (6088): No heartbeat from core client for 30 sec - exiting 09:19:38 (6088): No heartbeat from core client for 30 sec - exiting 09:19:39 (6088): No heartbeat from core client for 30 sec - exiting 09:19:40 (6088): No heartbeat from core client for 30 sec - exiting 09:19:41 (6088): No heartbeat from core client for 30 sec - exiting 09:19:43 (6088): No heartbeat from core client for 30 sec - exiting 09:19:44 (6088): No heartbeat from core client for 30 sec - exiting 09:19:45 (6088): No heartbeat from core client for 30 sec - exiting 09:19:46 (6088): No heartbeat from core client for 30 sec - exiting 09:19:47 (6088): No heartbeat from core client for 30 sec - exiting 09:19:48 (6088): No heartbeat from core client for 30 sec - exiting 09:19:49 (6088): No heartbeat from core client for 30 sec - exiting 09:19:50 (6088): No heartbeat from core client for 30 sec - exiting 09:19:51 (6088): No heartbeat from core client for 30 sec - exiting 09:19:52 (6088): No heartbeat from core client for 30 sec - exiting 09:19:53 (6088): No heartbeat from core client for 30 sec - exiting 09:19:55 (6088): No heartbeat from core client for 30 sec - exiting 09:19:56 (6088): No heartbeat from core client for 30 sec - exiting 09:19:57 (6088): No heartbeat from core client for 30 sec - exiting 09:19:58 (6088): No heartbeat from core client for 30 sec - exiting 09:19:59 (6088): No heartbeat from core client for 30 sec - exiting 09:20:00 (6088): No heartbeat from core client for 30 sec - exiting 09:20:01 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:52:31 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:04:38 (6604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4888, iMonCtr=1 Model crash detected, will try to restart... 22:16:03 (6800): No heartbeat from core client for 30 sec - exiting 22:16:04 (6800): No heartbeat from core client for 30 sec - exiting 22:16:05 (6800): No heartbeat from core client for 30 sec - exiting 22:16:06 (6800): No heartbeat from core client for 30 sec - exiting 22:16:08 (6800): No heartbeat from core client for 30 sec - exiting 22:16:09 (6800): No heartbeat from core client for 30 sec - exiting 22:16:10 (6800): No heartbeat from core client for 30 sec - exiting 22:16:11 (6800): No heartbeat from core client for 30 sec - exiting 22:16:12 (6800): No heartbeat from core client for 30 sec - exiting 22:16:13 (6800): No heartbeat from core client for 30 sec - exiting 22:16:14 (6800): No heartbeat from core client for 30 sec - exiting 22:16:15 (6800): No heartbeat from core client for 30 sec - exiting 22:16:16 (6800): No heartbeat from core client for 30 sec - exiting 22:16:17 (6800): No heartbeat from core client for 30 sec - exiting 22:16:18 (6800): No heartbeat from core client for 30 sec - exiting 22:16:19 (6800): No heartbeat from core client for 30 sec - exiting 22:16:20 (6800): No heartbeat from core client for 30 sec - exiting 22:16:21 (6800): No heartbeat from core client for 30 sec - exiting 22:16:22 (6800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:58 (5572): No heartbeat from core client for 30 sec - exiting 22:29:59 (5572): No heartbeat from core client for 30 sec - exiting 22:30:00 (5572): No heartbeat from core client for 30 sec - exiting 22:30:02 (5572): No heartbeat from core client for 30 sec - exiting 22:30:03 (5572): No heartbeat from core client for 30 sec - exiting 22:30:04 (5572): No heartbeat from core client for 30 sec - exiting 22:30:05 (5572): No heartbeat from core client for 30 sec - exiting 22:30:06 (5572): No heartbeat from core client for 30 sec - exiting 22:30:07 (5572): No heartbeat from core client for 30 sec - exiting 22:30:08 (5572): No heartbeat from core client for 30 sec - exiting 22:30:09 (5572): No heartbeat from core client for 30 sec - exiting 22:30:10 (5572): No heartbeat from core client for 30 sec - exiting 22:30:11 (5572): No heartbeat from core client for 30 sec - exiting 22:30:12 (5572): No heartbeat from core client for 30 sec - exiting 22:30:13 (5572): No heartbeat from core client for 30 sec - exiting 22:30:14 (5572): No heartbeat from core client for 30 sec - exiting 22:30:15 (5572): No heartbeat from core client for 30 sec - exiting 22:30:16 (5572): No heartbeat from core client for 30 sec - exiting 22:30:17 (5572): No heartbeat from core client for 30 sec - exiting 22:30:18 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:10 (6464): No heartbeat from core client for 30 sec - exiting 09:30:11 (6464): No heartbeat from core client for 30 sec - exiting 09:30:12 (6464): No heartbeat from core client for 30 sec - exiting 09:30:13 (6464): No heartbeat from core client for 30 sec - exiting 09:30:15 (6464): No heartbeat from core client for 30 sec - exiting 09:30:16 (6464): No heartbeat from core client for 30 sec - exiting 09:30:17 (6464): No heartbeat from core client for 30 sec - exiting 09:30:18 (6464): No heartbeat from core client for 30 sec - exiting 09:30:19 (6464): No heartbeat from core client for 30 sec - exiting 09:30:20 (6464): No heartbeat from core client for 30 sec - exiting 09:30:21 (6464): No heartbeat from core client for 30 sec - exiting 09:30:22 (6464): No heartbeat from core client for 30 sec - exiting 09:30:23 (6464): No heartbeat from core client for 30 sec - exiting 09:30:24 (6464): No heartbeat from core client for 30 sec - exiting 09:30:25 (6464): No heartbeat from core client for 30 sec - exiting 09:30:27 (6464): No heartbeat from core client for 30 sec - exiting 09:30:28 (6464): No heartbeat from core client for 30 sec - exiting 09:30:29 (6464): No heartbeat from core client for 30 sec - exiting 09:30:30 (6464): No heartbeat from core client for 30 sec - exiting 09:30:31 (6464): No heartbeat from core client for 30 sec - exiting 09:30:32 (6464): No heartbeat from core client for 30 sec - exiting 09:30:33 (6464): No heartbeat from core client for 30 sec - exiting 09:30:34 (6464): No heartbeat from core client for 30 sec - exiting 09:30:35 (6464): No heartbeat from core client for 30 sec - exiting 09:30:36 (6464): No heartbeat from core client for 30 sec - exiting 09:30:37 (6464): No heartbeat from core client for 30 sec - exiting 09:30:39 (6464): No heartbeat from core client for 30 sec - exiting 09:30:40 (6464): No heartbeat from core client for 30 sec - exiting 09:30:41 (6464): No heartbeat from core client for 30 sec - exiting 09:30:42 (6464): No heartbeat from core client for 30 sec - exiting 09:30:43 (6464): No heartbeat from core client for 30 sec - exiting 09:30:44 (6464): No heartbeat from core client for 30 sec - exiting 09:30:45 (6464): No heartbeat from core client for 30 sec - exiting 09:30:46 (6464): No heartbeat from core client for 30 sec - exiting 09:30:47 (6464): No heartbeat from core client for 30 sec - exiting 09:30:48 (6464): No heartbeat from core client for 30 sec - exiting 09:30:49 (6464): No heartbeat from core client for 30 sec - exiting 09:30:50 (6464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:51 (6464): No heartbeat from core client for 30 sec - exiting 09:30:52 (6464): No heartbeat from core client for 30 sec - exiting 15:16:07 (3448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774A71F3 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x004703A7 write attempt to address 0x007FD164 Engaging BOINC Windows Runtime Debugger... Signal 22 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Oct 2012 00:31:07	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	259,200	345,382	1.3325
12 Oct 2012 14:51:20	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	233,280	319,164	1.3682
12 Oct 2012 03:39:22	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	207,360	284,801	1.3735
10 Oct 2012 16:53:21	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	181,440	256,859	1.4157
10 Oct 2012 04:25:24	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	155,520	222,465	1.4305
09 Oct 2012 15:21:47	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	129,600	189,542	1.4625
08 Oct 2012 01:19:17	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	103,680	151,176	1.4581
07 Oct 2012 15:01:31	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	77,760	112,037	1.4408
05 Oct 2012 18:40:58	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	51,840	75,134	1.4493
05 Oct 2012 06:01:55	1238808	15330024	hadcm3n_zj6l_1880_40_008200984_2	25,920	40,896	1.5778