Task 16145838

Name	hadcm3n_81ox_1980_40_008460132_3
Workunit	8610988
Created	16 Dec 2013, 21:08:55 UTC
Sent	16 Dec 2013, 21:08:58 UTC
Report deadline	18 Mar 2014, 4:36:09 UTC
Received	27 Jan 2014, 17:55:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1166130
Run time	14 days 16 hours 55 min 27 sec
CPU time	13 days 2 hours 38 min 2 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11444, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3484, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1868, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:20:30 (7528): No heartbeat from core client for 30 sec - exiting 22:20:32 (7528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:09:16 (6044): No heartbeat from core client for 30 sec - exiting 00:09:17 (6044): No heartbeat from core client for 30 sec - exiting 00:09:18 (6044): No heartbeat from core client for 30 sec - exiting 00:09:20 (6044): No heartbeat from core client for 30 sec - exiting 00:09:21 (6044): No heartbeat from core client for 30 sec - exiting 00:09:22 (6044): No heartbeat from core client for 30 sec - exiting 00:09:23 (6044): No heartbeat from core client for 30 sec - exiting 00:09:24 (6044): No heartbeat from core client for 30 sec - exiting 00:09:25 (6044): No heartbeat from core client for 30 sec - exiting 00:09:26 (6044): No heartbeat from core client for 30 sec - exiting 00:09:27 (6044): No heartbeat from core client for 30 sec - exiting 00:09:28 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:32:29 (5212): No heartbeat from core client for 30 sec - exiting 00:32:30 (5212): No heartbeat from core client for 30 sec - exiting 00:32:31 (5212): No heartbeat from core client for 30 sec - exiting 00:32:32 (5212): No heartbeat from core client for 30 sec - exiting 00:32:33 (5212): No heartbeat from core client for 30 sec - exiting 00:32:35 (5212): No heartbeat from core client for 30 sec - exiting 00:32:36 (5212): No heartbeat from core client for 30 sec - exiting 00:32:37 (5212): No heartbeat from core client for 30 sec - exiting 00:32:38 (5212): No heartbeat from core client for 30 sec - exiting 00:32:39 (5212): No heartbeat from core client for 30 sec - exiting 00:32:40 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2040, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 01:42:35 (4344): No heartbeat from core client for 30 sec - exiting 01:42:37 (4344): No heartbeat from core client for 30 sec - exiting 01:42:38 (4344): No heartbeat from core client for 30 sec - exiting 01:42:39 (4344): No heartbeat from core client for 30 sec - exiting 01:42:40 (4344): No heartbeat from core client for 30 sec - exiting 01:42:41 (4344): No heartbeat from core client for 30 sec - exiting 01:42:42 (4344): No heartbeat from core client for 30 sec - exiting 01:42:43 (4344): No heartbeat from core client for 30 sec - exiting 01:42:44 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6676, iMonCtr=1 Model crash detected, will try to restart... 07:06:24 (8008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:29:53 (1836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:32:33 (7744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:34 (7744): No heartbeat from core client for 30 sec - exiting 04:32:35 (7744): No heartbeat from core client for 30 sec - exiting 04:32:36 (7744): No heartbeat from core client for 30 sec - exiting 04:32:37 (7744): No heartbeat from core client for 30 sec - exiting 04:32:38 (7744): No heartbeat from core client for 30 sec - exiting 04:32:40 (7744): No heartbeat from core client for 30 sec - exiting 04:32:41 (7744): No heartbeat from core client for 30 sec - exiting 04:32:42 (7744): No heartbeat from core client for 30 sec - exiting 04:32:43 (7744): No heartbeat from core client for 30 sec - exiting 04:32:44 (7744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:10:21 (8776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:38 (4556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:13:08 (6984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:36:10 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CE3AC3 read attempt to address 0x403CC806 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CE3AC3 read attempt to address 0x403CC806 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_81ox_1980_40_008460132/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Jan 2014 17:03:42	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	518,400	1,129,975	2.1797
25 Jan 2014 06:06:40	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	492,480	1,067,306	2.1672
24 Jan 2014 11:36:42	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	466,560	1,007,932	2.1603
21 Jan 2014 17:59:42	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	440,640	949,860	2.1556
19 Jan 2014 17:16:11	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	414,720	894,972	2.1580
18 Jan 2014 15:55:01	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	388,800	838,372	2.1563
15 Jan 2014 14:58:24	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	362,880	795,275	2.1916
11 Jan 2014 23:36:15	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	336,960	740,119	2.1965
09 Jan 2014 19:23:05	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	311,040	684,265	2.1999
07 Jan 2014 21:22:18	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	285,120	642,949	2.2550
05 Jan 2014 21:20:23	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	259,200	591,266	2.2811
03 Jan 2014 05:34:56	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	233,280	533,141	2.2854
01 Jan 2014 23:12:51	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	207,360	476,905	2.2999
01 Jan 2014 04:03:11	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	181,440	419,778	2.3136
31 Dec 2013 10:20:08	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	155,520	361,910	2.3271
30 Dec 2013 15:13:51	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	129,600	305,543	2.3576
29 Dec 2013 15:33:42	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	103,680	246,534	2.3778
26 Dec 2013 23:09:33	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	77,760	184,326	2.3704
24 Dec 2013 19:17:01	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	51,840	121,450	2.3428
19 Dec 2013 20:25:14	1166130	16145838	hadcm3n_81ox_1980_40_008460132_3	25,920	59,644	2.3011