Task 16032198

Name	hadcm3n_o3jp_1980_40_008407857_1
Workunit	8558713
Created	23 Sep 2013, 17:25:05 UTC
Sent	23 Sep 2013, 17:30:36 UTC
Report deadline	24 Dec 2013, 0:57:47 UTC
Received	13 Nov 2013, 15:09:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1046856
Run time	23 days 7 hours 38 min 13 sec
CPU time	23 days 7 hours 38 min 13 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	0.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3188, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2792, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4712, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4892, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4092, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 09:57:44 (3864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3976, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=324, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on o3jpko.dai7am0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1096, iMonCtr=1 Model crash detected, will try to restart... 12:41:27 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3jpko.dak05f0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1628, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on o3jpko.dak38t0 Ocean Restart file copy failed on o3jpko.dak4bq0 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on o3jpko.dak6b80 Ocean Restart file copy failed on o3jpko.dak6b90 Ocean Restart file copy failed on o3jpko.dak6ba0 CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3jpko.dak84h0 Ocean Restart file copy failed on o3jpko.dak84i0 Ocean Restart file copy failed on o3jpko.dak84j0 Ocean Restart file copy failed on o3jpko.dak84k0 Ocean Restart file copy failed on o3jpko.dak84l0 Ocean Restart file copy failed on o3jpko.dak84m0 Ocean Restart file copy failed on o3jpko.dak84n0 Ocean Restart file copy failed on o3jpko.dak84o0 Ocean Restart file copy failed on o3jpko.dak84p0 Ocean Restart file copy failed on o3jpko.dak84q0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5052, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on o3jpko.dal72u0 Ocean Restart file copy failed on o3jpko.dal74u0 Ocean Restart file copy failed on o3jpko.dal7510 Ocean Restart file copy failed on o3jpko.dal7520 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4268, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on o3jpko.dal93c0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3jpko.dal99l0 Ocean Restart file copy failed on o3jpko.dal99m0 Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77D73AC3 read attempt to address 0x403715C5 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B73AC3 read attempt to address 0x403715C5 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o3jp_1980_40_008407857/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Nov 2013 11:36:56	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	1,036,800	2,014,033	1.9425
09 Nov 2013 17:29:13	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	1,010,880	1,977,320	1.9560
09 Nov 2013 07:15:49	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	984,960	1,940,806	1.9704
08 Nov 2013 19:32:49	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	959,040	1,899,190	1.9803
07 Nov 2013 18:48:08	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	933,120	1,859,655	1.9929
06 Nov 2013 17:45:31	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	907,200	1,823,162	2.0097
05 Nov 2013 16:10:47	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	881,280	1,786,686	2.0274
03 Nov 2013 21:04:55	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	855,360	1,750,006	2.0459
03 Nov 2013 03:16:27	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	829,440	1,712,067	2.0641
02 Nov 2013 08:18:03	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	803,520	1,674,606	2.0841
31 Oct 2013 12:50:11	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	777,600	1,634,776	2.1023
30 Oct 2013 06:06:44	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	751,680	1,598,221	2.1262
26 Oct 2013 15:44:36	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	725,760	1,561,896	2.1521
25 Oct 2013 23:32:48	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	699,840	1,523,358	2.1767
25 Oct 2013 12:00:19	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	673,920	1,482,703	2.2001
24 Oct 2013 17:24:30	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	648,000	1,442,586	2.2262
23 Oct 2013 10:17:09	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	622,080	1,405,802	2.2598
22 Oct 2013 14:44:25	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	596,160	1,369,874	2.2978
21 Oct 2013 14:17:19	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	570,240	1,334,629	2.3405
21 Oct 2013 04:17:21	1046856	16032198	hadcm3n_o3jp_1980_40_008407857_1	544,320	1,298,777	2.3861