Task 16221156

Name	hadcm3n_7bv6_1980_40_008426661_2
Workunit	8577517
Created	14 Jan 2014, 1:09:38 UTC
Sent	14 Jan 2014, 1:37:18 UTC
Report deadline	15 Apr 2014, 9:04:29 UTC
Received	10 Feb 2014, 14:09:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1158872
Run time	19 days 17 hours 44 min 57 sec
CPU time	18 days 3 hours 33 min 37 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	0.84 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12164, iMonCtr=1 Model crash detected, will try to restart... 07:01:58 (4116): No heartbeat from core client for 30 sec - exiting 07:01:59 (4116): No heartbeat from core client for 30 sec - exiting 07:02:01 (4116): No heartbeat from core client for 30 sec - exiting 07:02:02 (4116): No heartbeat from core client for 30 sec - exiting 07:02:03 (4116): No heartbeat from core client for 30 sec - exiting 07:02:04 (4116): No heartbeat from core client for 30 sec - exiting 07:02:05 (4116): No heartbeat from core client for 30 sec - exiting 07:02:06 (4116): No heartbeat from core client for 30 sec - exiting 07:02:07 (4116): No heartbeat from core client for 30 sec - exiting 07:02:08 (4116): No heartbeat from core client for 30 sec - exiting 07:02:09 (4116): No heartbeat from core client for 30 sec - exiting 07:02:10 (4116): No heartbeat from core client for 30 sec - exiting 07:02:11 (4116): No heartbeat from core client for 30 sec - exiting 07:02:13 (4116): No heartbeat from core client for 30 sec - exiting 07:02:14 (4116): No heartbeat from core client for 30 sec - exiting 07:02:15 (4116): No heartbeat from core client for 30 sec - exiting 07:02:16 (4116): No heartbeat from core client for 30 sec - exiting 07:02:17 (4116): No heartbeat from core client for 30 sec - exiting 07:02:18 (4116): No heartbeat from core client for 30 sec - exiting 07:02:19 (4116): No heartbeat from core client for 30 sec - exiting 07:02:20 (4116): No heartbeat from core client for 30 sec - exiting 07:02:21 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:33:04 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:57:16 (9776): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:57:17 (9776): No heartbeat from core client for 30 sec - exiting 07:57:18 (9776): No heartbeat from core client for 30 sec - exiting 07:57:19 (9776): No heartbeat from core client for 30 sec - exiting 07:57:21 (9776): No heartbeat from core client for 30 sec - exiting 07:57:22 (9776): No heartbeat from core client for 30 sec - exiting 07:57:23 (9776): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:53:43 (10516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:12:08 (16928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:09 (16928): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:51:07 (5708): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:51:08 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:44:02 (11140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:38:17 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:19 (4684): No heartbeat from core client for 30 sec - exiting 06:45:06 (12076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:19:20 (7584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:19:21 (7584): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 13:50:41 (7252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:05 (17228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:14 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:24:31 (17872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:39 (11436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:21 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:39:20 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:59 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:21 (11200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:36 (9768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:45 (16648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:41 (17812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:31 (10448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:29:22 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:28 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:36:47 (13748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:54 (10112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:19 (11728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:32 (17860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:28:24 (12620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:31:32 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:08:20 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:49 (6556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:53 (6556): No heartbeat from core client for 30 sec - exiting 13:51:42 (12396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:45:07 (18056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:09 (18056): No heartbeat from core client for 30 sec - exiting 03:10:02 (17428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:24 (16976): No heartbeat from core client for 30 sec - exiting 03:11:25 (16976): No heartbeat from core client for 30 sec - exiting 03:11:26 (16976): No heartbeat from core client for 30 sec - exiting 03:11:27 (16976): No heartbeat from core client for 30 sec - exiting 03:11:28 (16976): No heartbeat from core client for 30 sec - exiting 03:11:29 (16976): No heartbeat from core client for 30 sec - exiting 03:11:30 (16976): No heartbeat from core client for 30 sec - exiting 03:11:31 (16976): No heartbeat from core client for 30 sec - exiting 03:11:32 (16976): No heartbeat from core client for 30 sec - exiting 03:11:33 (16976): No heartbeat from core client for 30 sec - exiting 03:11:34 (16976): No heartbeat from core client for 30 sec - exiting 03:11:35 (16976): No heartbeat from core client for 30 sec - exiting 03:11:36 (16976): No heartbeat from core client for 30 sec - exiting 03:11:37 (16976): No heartbeat from core client for 30 sec - exiting 03:11:38 (16976): No heartbeat from core client for 30 sec - exiting 03:11:39 (16976): No heartbeat from core client for 30 sec - exiting 03:11:40 (16976): No heartbeat from core client for 30 sec - exiting 03:11:41 (16976): No heartbeat from core client for 30 sec - exiting 03:11:42 (16976): No heartbeat from core client for 30 sec - exiting 03:11:43 (16976): No heartbeat from core client for 30 sec - exiting 03:11:44 (16976): No heartbeat from core client for 30 sec - exiting 03:11:45 (16976): No heartbeat from core client for 30 sec - exiting 03:11:46 (16976): No heartbeat from core client for 30 sec - exiting 03:11:47 (16976): No heartbeat from core client for 30 sec - exiting 03:11:48 (16976): No heartbeat from core client for 30 sec - exiting 03:11:49 (16976): No heartbeat from core client for 30 sec - exiting 03:11:50 (16976): No heartbeat from core client for 30 sec - exiting 03:11:51 (16976): No heartbeat from core client for 30 sec - exiting 03:11:52 (16976): No heartbeat from core client for 30 sec - exiting 03:11:53 (16976): No heartbeat from core client for 30 sec - exiting 03:11:54 (16976): No heartbeat from core client for 30 sec - exiting 03:11:55 (16976): No heartbeat from core client for 30 sec - exiting 03:11:56 (16976): No heartbeat from core client for 30 sec - exiting 03:11:57 (16976): No heartbeat from core client for 30 sec - exiting 03:11:58 (16976): No heartbeat from core client for 30 sec - exiting 03:11:59 (16976): No heartbeat from core client for 30 sec - exiting 03:12:00 (16976): No heartbeat from core client for 30 sec - exiting 03:12:01 (16976): No heartbeat from core client for 30 sec - exiting 03:12:02 (16976): No heartbeat from core client for 30 sec - exiting 03:12:03 (16976): No heartbeat from core client for 30 sec - exiting 03:12:04 (16976): No heartbeat from core client for 30 sec - exiting 03:12:06 (16976): No heartbeat from core client for 30 sec - exiting 03:12:07 (16976): No heartbeat from core client for 30 sec - exiting 03:12:08 (16976): No heartbeat from core client for 30 sec - exiting 03:12:09 (16976): No heartbeat from core client for 30 sec - exiting 03:12:10 (16976): No heartbeat from core client for 30 sec - exiting 03:12:11 (16976): No heartbeat from core client for 30 sec - exiting 03:12:12 (16976): No heartbeat from core client for 30 sec - exiting 03:12:13 (16976): No heartbeat from core client for 30 sec - exiting 03:12:14 (16976): No heartbeat from core client for 30 sec - exiting 03:12:15 (16976): No heartbeat from core client for 30 sec - exiting 03:12:16 (16976): No heartbeat from core client for 30 sec - exiting 03:12:17 (16976): No heartbeat from core client for 30 sec - exiting 03:12:18 (16976): No heartbeat from core client for 30 sec - exiting 03:12:19 (16976): No heartbeat from core client for 30 sec - exiting 03:12:20 (16976): No heartbeat from core client for 30 sec - exiting 03:12:21 (16976): No heartbeat from core client for 30 sec - exiting 03:12:22 (16976): No heartbeat from core client for 30 sec - exiting 03:12:23 (16976): No heartbeat from core client for 30 sec - exiting 03:12:24 (16976): No heartbeat from core client for 30 sec - exiting 03:12:25 (16976): No heartbeat from core client for 30 sec - exiting 03:12:26 (16976): No heartbeat from core client for 30 sec - exiting 03:12:27 (16976): No heartbeat from core client for 30 sec - exiting 03:12:28 (16976): No heartbeat from core client for 30 sec - exiting 03:12:29 (16976): No heartbeat from core client for 30 sec - exiting 03:12:30 (16976): No heartbeat from core client for 30 sec - exiting 03:12:31 (16976): No heartbeat from core client for 30 sec - exiting 03:12:32 (16976): No heartbeat from core client for 30 sec - exiting 03:12:33 (16976): No heartbeat from core client for 30 sec - exiting 03:12:34 (16976): No heartbeat from core client for 30 sec - exiting 03:12:35 (16976): No heartbeat from core client for 30 sec - exiting 03:12:36 (16976): No heartbeat from core client for 30 sec - exiting 03:12:37 (16976): No heartbeat from core client for 30 sec - exiting 03:12:38 (16976): No heartbeat from core client for 30 sec - exiting 03:12:39 (16976): No heartbeat from core client for 30 sec - exiting 03:12:40 (16976): No heartbeat from core client for 30 sec - exiting 03:12:41 (16976): No heartbeat from core client for 30 sec - exiting 03:12:42 (16976): No heartbeat from core client for 30 sec - exiting 03:12:43 (16976): No heartbeat from core client for 30 sec - exiting 03:12:44 (16976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77BB3AC3 read attempt to address 0x40C46302 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77BB3AC3 read attempt to address 0x40C46302 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\Program Files\BOINC\Data/projects/climateprediction.net/hadcm3n_7bv6_1980_40_008426661/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Feb 2014 18:18:48	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	259,200	1,521,422	5.8697
07 Feb 2014 21:28:23	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	233,280	1,364,981	5.8513
05 Feb 2014 11:06:20	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	207,360	1,205,257	5.8124
03 Feb 2014 02:24:59	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	181,440	1,048,563	5.7791
01 Feb 2014 00:30:18	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	155,520	897,087	5.7683
30 Jan 2014 23:55:54	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	129,600	750,192	5.7885
26 Jan 2014 14:33:10	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	103,680	611,996	5.9027
23 Jan 2014 10:20:37	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	77,760	467,971	6.0181
18 Jan 2014 13:13:26	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	51,840	301,785	5.8215
16 Jan 2014 15:34:49	1158872	16221156	hadcm3n_7bv6_1980_40_008426661_2	25,920	149,218	5.7569