Task 15926668

Name	hadcm3n_3df4_1980_40_008405309_0
Workunit	8556165
Created	20 Aug 2013, 2:09:27 UTC
Sent	20 Aug 2013, 2:10:33 UTC
Report deadline	19 Nov 2013, 9:37:44 UTC
Received	13 Oct 2013, 15:54:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1245322
Run time	5 days 7 hours 10 min 4 sec
CPU time	4 days 16 hours 19 min 47 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.39 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 16:28:35 (1096): No heartbeat from core client for 30 sec - exiting 16:28:36 (1096): No heartbeat from core client for 30 sec - exiting 16:28:37 (1096): No heartbeat from core client for 30 sec - exiting 16:28:38 (1096): No heartbeat from core client for 30 sec - exiting 16:28:39 (1096): No heartbeat from core client for 30 sec - exiting 16:28:40 (1096): No heartbeat from core client for 30 sec - exiting 16:28:41 (1096): No heartbeat from core client for 30 sec - exiting 16:28:42 (1096): No heartbeat from core client for 30 sec - exiting 16:28:43 (1096): No heartbeat from core client for 30 sec - exiting 16:28:44 (1096): No heartbeat from core client for 30 sec - exiting 16:28:46 (1096): No heartbeat from core client for 30 sec - exiting 16:28:47 (1096): No heartbeat from core client for 30 sec - exiting 16:28:48 (1096): No heartbeat from core client for 30 sec - exiting 16:28:49 (1096): No heartbeat from core client for 30 sec - exiting 16:28:50 (1096): No heartbeat from core client for 30 sec - exiting 16:28:51 (1096): No heartbeat from core client for 30 sec - exiting 16:28:52 (1096): No heartbeat from core client for 30 sec - exiting 16:28:53 (1096): No heartbeat from core client for 30 sec - exiting 16:28:54 (1096): No heartbeat from core client for 30 sec - exiting 16:28:55 (1096): No heartbeat from core client for 30 sec - exiting 16:28:57 (1096): No heartbeat from core client for 30 sec - exiting 16:28:58 (1096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4584, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:15:01 (5984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7296, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 20:34:07 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:48:01 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:05:14 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:48:04 (2596): No heartbeat from core client for 30 sec - exiting 16:48:05 (2596): No heartbeat from core client for 30 sec - exiting 16:48:06 (2596): No heartbeat from core client for 30 sec - exiting 16:48:07 (2596): No heartbeat from core client for 30 sec - exiting 16:48:08 (2596): No heartbeat from core client for 30 sec - exiting 16:48:09 (2596): No heartbeat from core client for 30 sec - exiting 16:48:10 (2596): No heartbeat from core client for 30 sec - exiting 16:48:11 (2596): No heartbeat from core client for 30 sec - exiting 16:48:13 (2596): No heartbeat from core client for 30 sec - exiting 16:48:14 (2596): No heartbeat from core client for 30 sec - exiting 16:48:15 (2596): No heartbeat from core client for 30 sec - exiting 16:48:16 (2596): No heartbeat from core client for 30 sec - exiting 16:48:17 (2596): No heartbeat from core client for 30 sec - exiting 16:48:18 (2596): No heartbeat from core client for 30 sec - exiting 16:48:19 (2596): No heartbeat from core client for 30 sec - exiting 16:48:20 (2596): No heartbeat from core client for 30 sec - exiting 16:48:21 (2596): No heartbeat from core client for 30 sec - exiting 16:48:22 (2596): No heartbeat from core client for 30 sec - exiting 16:48:23 (2596): No heartbeat from core client for 30 sec - exiting 16:48:25 (2596): No heartbeat from core client for 30 sec - exiting 16:48:26 (2596): No heartbeat from core client for 30 sec - exiting 16:48:27 (2596): No heartbeat from core client for 30 sec - exiting 16:48:28 (2596): No heartbeat from core client for 30 sec - exiting 16:48:29 (2596): No heartbeat from core client for 30 sec - exiting 16:48:30 (2596): No heartbeat from core client for 30 sec - exiting 16:48:31 (2596): No heartbeat from core client for 30 sec - exiting 16:48:32 (2596): No heartbeat from core client for 30 sec - exiting 16:48:33 (2596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:05:54 (6180): No heartbeat from core client for 30 sec - exiting 16:05:55 (6180): No heartbeat from core client for 30 sec - exiting 16:05:56 (6180): No heartbeat from core client for 30 sec - exiting 16:05:57 (6180): No heartbeat from core client for 30 sec - exiting 16:05:58 (6180): No heartbeat from core client for 30 sec - exiting 16:05:59 (6180): No heartbeat from core client for 30 sec - exiting 16:06:00 (6180): No heartbeat from core client for 30 sec - exiting 16:06:01 (6180): No heartbeat from core client for 30 sec - exiting 16:06:02 (6180): No heartbeat from core client for 30 sec - exiting 16:06:03 (6180): No heartbeat from core client for 30 sec - exiting 16:06:04 (6180): No heartbeat from core client for 30 sec - exiting 16:06:05 (6180): No heartbeat from core client for 30 sec - exiting 16:06:06 (6180): No heartbeat from core client for 30 sec - exiting 16:06:07 (6180): No heartbeat from core client for 30 sec - exiting 16:06:08 (6180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:24:51 (6224): No heartbeat from core client for 30 sec - exiting 16:24:52 (6224): No heartbeat from core client for 30 sec - exiting 16:24:53 (6224): No heartbeat from core client for 30 sec - exiting 16:24:54 (6224): No heartbeat from core client for 30 sec - exiting 16:24:55 (6224): No heartbeat from core client for 30 sec - exiting 16:24:56 (6224): No heartbeat from core client for 30 sec - exiting 16:24:57 (6224): No heartbeat from core client for 30 sec - exiting 16:24:58 (6224): No heartbeat from core client for 30 sec - exiting 16:24:59 (6224): No heartbeat from core client for 30 sec - exiting 16:25:00 (6224): No heartbeat from core client for 30 sec - exiting 16:25:01 (6224): No heartbeat from core client for 30 sec - exiting 16:25:02 (6224): No heartbeat from core client for 30 sec - exiting 16:25:03 (6224): No heartbeat from core client for 30 sec - exiting 16:25:04 (6224): No heartbeat from core client for 30 sec - exiting 16:25:05 (6224): No heartbeat from core client for 30 sec - exiting 16:25:06 (6224): No heartbeat from core client for 30 sec - exiting 16:25:07 (6224): No heartbeat from core client for 30 sec - exiting 16:25:08 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:12:09 (1832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=132, iMonCtr=1 Model crash detected, will try to restart... 16:10:48 (4896): No heartbeat from core client for 30 sec - exiting 16:10:49 (4896): No heartbeat from core client for 30 sec - exiting 16:10:50 (4896): No heartbeat from core client for 30 sec - exiting 16:10:51 (4896): No heartbeat from core client for 30 sec - exiting 16:10:52 (4896): No heartbeat from core client for 30 sec - exiting 16:10:53 (4896): No heartbeat from core client for 30 sec - exiting 16:10:54 (4896): No heartbeat from core client for 30 sec - exiting 16:10:55 (4896): No heartbeat from core client for 30 sec - exiting 16:10:56 (4896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:20:27 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:22:23 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:20 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10064, iMonCtr=1 Model crash detected, will try to restart... 16:25:17 (6200): No heartbeat from core client for 30 sec - exiting 16:25:18 (6200): No heartbeat from core client for 30 sec - exiting 16:25:19 (6200): No heartbeat from core client for 30 sec - exiting 16:25:20 (6200): No heartbeat from core client for 30 sec - exiting 16:25:21 (6200): No heartbeat from core client for 30 sec - exiting 16:25:22 (6200): No heartbeat from core client for 30 sec - exiting 16:25:23 (6200): No heartbeat from core client for 30 sec - exiting 16:25:24 (6200): No heartbeat from core client for 30 sec - exiting 16:25:25 (6200): No heartbeat from core client for 30 sec - exiting 16:25:26 (6200): No heartbeat from core client for 30 sec - exiting 16:25:27 (6200): No heartbeat from core client for 30 sec - exiting 16:25:28 (6200): No heartbeat from core client for 30 sec - exiting 16:25:29 (6200): No heartbeat from core client for 30 sec - exiting 16:25:30 (6200): No heartbeat from core client for 30 sec - exiting 16:25:31 (6200): No heartbeat from core client for 30 sec - exiting 16:25:32 (6200): No heartbeat from core client for 30 sec - exiting 16:25:33 (6200): No heartbeat from core client for 30 sec - exiting 16:25:34 (6200): No heartbeat from core client for 30 sec - exiting 16:25:35 (6200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:31:44 (7088): No heartbeat from core client for 30 sec - exiting 16:31:45 (7088): No heartbeat from core client for 30 sec - exiting 16:31:46 (7088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:25:54 (3700): No heartbeat from core client for 30 sec - exiting 16:25:55 (3700): No heartbeat from core client for 30 sec - exiting 16:25:56 (3700): No heartbeat from core client for 30 sec - exiting 16:25:57 (3700): No heartbeat from core client for 30 sec - exiting 16:25:58 (3700): No heartbeat from core client for 30 sec - exiting 16:25:59 (3700): No heartbeat from core client for 30 sec - exiting 16:26:00 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:46 (2548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77DD7383 read attempt to address 0x40A11833 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771C3AC3 read attempt to address 0x40A1183B Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3df4_1980_40_008405309/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Oct 2013 04:52:57	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	259,200	397,473	1.5335
12 Oct 2013 18:13:00	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	233,280	358,941	1.5387
08 Oct 2013 23:03:03	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	207,360	320,183	1.5441
06 Oct 2013 16:32:47	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	181,440	279,653	1.5413
29 Sep 2013 16:00:58	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	155,520	239,952	1.5429
19 Sep 2013 23:30:48	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	129,600	200,109	1.5441
08 Sep 2013 15:49:05	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	103,680	160,115	1.5443
06 Sep 2013 21:56:19	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	77,760	120,227	1.5461
31 Aug 2013 13:37:09	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	51,840	79,987	1.5430
25 Aug 2013 22:40:58	1245322	15926668	hadcm3n_3df4_1980_40_008405309_0	25,920	40,512	1.5630