Task 13548261

Name	hadcm3n_y9vt_1900_40_007521286_1
Workunit	7718761
Created	28 Oct 2011, 13:12:10 UTC
Sent	2 Nov 2011, 19:15:16 UTC
Report deadline	2 Feb 2012, 2:42:27 UTC
Received	20 Feb 2012, 17:26:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1177889
Run time	6 days 18 hours 16 min 36 sec
CPU time	6 days 0 hours 35 min 6 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:10:25 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=1 Model crash detected, will try to restart... 23:02:55 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4712, iMonCtr=1 Model crash detected, will try to restart... 21:13:58 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:40:50 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:40:59 (3420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4356, iMonCtr=1 Model crash detected, will try to restart... 15:19:12 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4336, iMonCtr=1 Model crash detected, will try to restart... 10:27:22 (1924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:50 (4572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5148, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3744, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4792, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3132, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:47:14 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:15 (6128): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/y9vtko.pja5c10 Error converting file to netcdf: dataout/y9vtko.pia5c10 Error converting file to netcdf: dataout/y9vtko.pfa5c10 Error converting file to netcdf: dataout/y9vtka.pha5c10 Error converting file to netcdf: dataout/y9vtka.pga5c10 Error converting file to netcdf: dataout/y9vtka.pea5c10 Error converting file to netcdf: dataout/y9vtka.pda5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3744, iMonCtr=1 Model crash detected, will try to restart... 09:40:07 (4152): No heartbeat from core client for 30 sec - exiting 09:40:08 (4152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:28:40 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:52:43 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:47:13 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:37:08 (4792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5996, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:59:28 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5828, iMonCtr=1 Model crash detected, will try to restart... 19:39:57 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:43 (5888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:08 (4464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1 Model crash detected, will try to restart... 21:06:38 (4728): No heartbeat from core client for 30 sec - exiting 21:06:39 (4728): No heartbeat from core client for 30 sec - exiting 21:06:40 (4728): No heartbeat from core client for 30 sec - exiting 21:06:41 (4728): No heartbeat from core client for 30 sec - exiting 21:06:42 (4728): No heartbeat from core client for 30 sec - exiting 21:06:44 (4728): No heartbeat from core client for 30 sec - exiting 21:06:45 (4728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:51:24 (4460): No heartbeat from core client for 30 sec - exiting 22:51:26 (4460): No heartbeat from core client for 30 sec - exiting 22:51:27 (4460): No heartbeat from core client for 30 sec - exiting 22:51:28 (4460): No heartbeat from core client for 30 sec - exiting 22:51:29 (4460): No heartbeat from core client for 30 sec - exiting 22:51:30 (4460): No heartbeat from core client for 30 sec - exiting 22:51:31 (4460): No heartbeat from core client for 30 sec - exiting 22:51:32 (4460): No heartbeat from core client for 30 sec - exiting 22:51:33 (4460): No heartbeat from core client for 30 sec - exiting 22:51:34 (4460): No heartbeat from core client for 30 sec - exiting 22:51:36 (4460): No heartbeat from core client for 30 sec - exiting 22:51:37 (4460): No heartbeat from core client for 30 sec - exiting 22:51:38 (4460): No heartbeat from core client for 30 sec - exiting 22:51:39 (4460): No heartbeat from core client for 30 sec - exiting 22:51:40 (4460): No heartbeat from core client for 30 sec - exiting 22:51:41 (4460): No heartbeat from core client for 30 sec - exiting 22:51:42 (4460): No heartbeat from core client for 30 sec - exiting 22:51:43 (4460): No heartbeat from core client for 30 sec - exiting 22:51:44 (4460): No heartbeat from core client for 30 sec - exiting 22:51:45 (4460): No heartbeat from core client for 30 sec - exiting 22:51:47 (4460): No heartbeat from core client for 30 sec - exiting 22:51:48 (4460): No heartbeat from core client for 30 sec - exiting 22:51:49 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:28 (4844): No heartbeat from core client for 30 sec - exiting 17:39:29 (4844): No heartbeat from core client for 30 sec - exiting 17:39:30 (4844): No heartbeat from core client for 30 sec - exiting 17:39:31 (4844): No heartbeat from core client for 30 sec - exiting 17:39:33 (4844): No heartbeat from core client for 30 sec - exiting 17:39:34 (4844): No heartbeat from core client for 30 sec - exiting 17:39:35 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... 09:52:28 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77163AB3 read attempt to address 0x4098759C Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77003AB3 read attempt to address 0x4098759C Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Feb 2012 05:42:35	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	259,200	518,607	2.0008
15 Feb 2012 05:01:50	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	233,280	465,694	1.9963
09 Feb 2012 06:38:29	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	207,360	412,640	1.9900
05 Feb 2012 08:09:01	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	181,440	358,778	1.9774
21 Jan 2012 04:34:38	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	155,520	299,456	1.9255
10 Jan 2012 02:21:56	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	129,600	247,195	1.9074
26 Dec 2011 07:09:50	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	103,680	196,049	1.8909
14 Dec 2011 07:52:28	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	77,760	143,106	1.8404
30 Nov 2011 04:05:08	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	51,840	94,204	1.8172
21 Nov 2011 08:08:16	1177889	13548261	hadcm3n_y9vt_1900_40_007521286_1	25,920	49,198	1.8981