Task 15900627

Name	hadcm3n_o5vt_1980_40_008388710_1
Workunit	8539569
Created	22 Jul 2013, 15:26:58 UTC
Sent	22 Jul 2013, 21:21:22 UTC
Report deadline	22 Oct 2013, 4:48:33 UTC
Received	28 Jan 2014, 2:37:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1281494
Run time	16 days 18 hours 57 min 14 sec
CPU time	16 days 10 hours 46 min 12 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> 17:23:36 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5260, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6116, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5968, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5540, iMonCtr=1 Model crash detected, will try to restart... 17:24:17 (1348): No heartbeat from core client for 30 sec - exiting 17:24:18 (1348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4740, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 13:49:41 (4912): No heartbeat from core client for 30 sec - exiting 13:49:42 (4912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5772, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2800, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5960, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5624, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5156, iMonCtr=1 Model crash detected, will try to restart... 15:20:29 (5356): No heartbeat from core client for 30 sec - exiting 15:20:30 (5356): No heartbeat from core client for 30 sec - exiting 15:20:31 (5356): No heartbeat from core client for 30 sec - exiting 15:20:32 (5356): No heartbeat from core client for 30 sec - exiting 15:20:33 (5356): No heartbeat from core client for 30 sec - exiting 15:20:34 (5356): No heartbeat from core client for 30 sec - exiting 15:20:35 (5356): No heartbeat from core client for 30 sec - exiting 15:20:37 (5356): No heartbeat from core client for 30 sec - exiting 15:20:38 (5356): No heartbeat from core client for 30 sec - exiting 15:20:39 (5356): No heartbeat from core client for 30 sec - exiting 15:20:40 (5356): No heartbeat from core client for 30 sec - exiting 15:20:41 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:44:49 (5524): No heartbeat from core client for 30 sec - exiting 17:44:50 (5524): No heartbeat from core client for 30 sec - exiting 17:44:51 (5524): No heartbeat from core client for 30 sec - exiting 17:44:52 (5524): No heartbeat from core client for 30 sec - exiting 17:44:54 (5524): No heartbeat from core client for 30 sec - exiting 17:44:55 (5524): No heartbeat from core client for 30 sec - exiting 17:44:56 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:05:18 (4352): No heartbeat from core client for 30 sec - exiting 13:05:19 (4352): No heartbeat from core client for 30 sec - exiting 13:05:20 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:41 (5116): No heartbeat from core client for 30 sec - exiting 17:21:42 (5116): No heartbeat from core client for 30 sec - exiting 17:21:44 (5116): No heartbeat from core client for 30 sec - exiting 17:21:45 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4192, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:28:42 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:39:22 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:40 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2728, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2728, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o5vtko.pjl4c10 Error converting file to netcdf: dataout/o5vtko.pil4c10 Error converting file to netcdf: dataout/o5vtko.pfl4c10 Error converting file to netcdf: dataout/o5vtka.phl4c10 Error converting file to netcdf: dataout/o5vtka.pgl4c10 Error converting file to netcdf: dataout/o5vtka.pel4c10 Error converting file to netcdf: dataout/o5vtka.pdl4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F537A2 read attempt to address 0x40CE3460 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Jan 2014 02:40:04	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	1,036,800	1,421,170	1.3707
27 Jan 2014 02:46:35	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	1,010,880	1,396,889	1.3819
26 Jan 2014 19:14:53	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	984,960	1,372,346	1.3933
25 Jan 2014 23:10:11	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	959,040	1,347,936	1.4055
25 Jan 2014 00:45:42	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	933,120	1,323,541	1.4184
14 Jan 2014 03:22:07	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	907,200	1,299,249	1.4322
12 Jan 2014 20:50:02	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	881,280	1,274,835	1.4466
11 Jan 2014 02:09:50	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	855,360	1,250,519	1.4620
07 Jan 2014 02:26:22	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	829,440	1,226,254	1.4784
21 Dec 2013 04:25:19	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	803,520	1,201,586	1.4954
17 Dec 2013 01:33:50	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	777,600	1,176,693	1.5132
15 Dec 2013 02:34:50	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	751,680	1,148,416	1.5278
09 Dec 2013 02:32:45	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	725,760	1,120,233	1.5435
07 Dec 2013 03:13:46	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	699,840	1,092,603	1.5612
26 Nov 2013 01:48:32	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	673,920	1,064,637	1.5798
05 Nov 2013 03:26:32	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	648,000	1,036,489	1.5995
03 Nov 2013 20:34:47	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	622,080	1,008,342	1.6209
02 Nov 2013 21:24:00	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	596,160	980,380	1.6445
28 Oct 2013 01:42:30	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	570,240	948,028	1.6625
23 Oct 2013 21:22:38	1281494	15900627	hadcm3n_o5vt_1980_40_008388710_1	544,320	907,611	1.6674