Task 16062329

Name	hadcm3n_o8ee_1900_40_008466009_2
Workunit	8616848
Created	9 Oct 2013, 2:02:55 UTC
Sent	9 Oct 2013, 2:33:03 UTC
Report deadline	8 Jan 2014, 10:00:14 UTC
Received	1 Jan 2014, 15:16:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1179495
Run time	45 days 2 hours 42 min 10 sec
CPU time	35 days 20 hours 4 min 55 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.47 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 04:37:39 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5436, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2256, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2256, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3732, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3732, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3732, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2744, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3732, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3588, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4184, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o8eeko.pja9c10 Error converting file to netcdf: dataout/o8eeko.pia9c10 Error converting file to netcdf: dataout/o8eeko.pfa9c10 Error converting file to netcdf: dataout/o8eeka.pha9c10 Error converting file to netcdf: dataout/o8eeka.pga9c10 Error converting file to netcdf: dataout/o8eeka.pea9c10 Error converting file to netcdf: dataout/o8eeka.pda9c10 Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3792, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:53:08 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1468, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5800, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... 16:47:09 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3376, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3644, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3820, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3896, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3896, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3896, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on o8eeko.dac8au0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2728, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3836, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3836, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3836, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3836, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3600, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3600, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3596, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 04:22:15 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4708, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:19:10 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:37:19 (7972): No heartbeat from core client for 30 sec - exiting 20:37:20 (7972): No heartbeat from core client for 30 sec - exiting 20:37:21 (7972): No heartbeat from core client for 30 sec - exiting 20:37:22 (7972): No heartbeat from core client for 30 sec - exiting 20:37:23 (7972): No heartbeat from core client for 30 sec - exiting 20:37:24 (7972): No heartbeat from core client for 30 sec - exiting 20:37:25 (7972): No heartbeat from core client for 30 sec - exiting 20:37:26 (7972): No heartbeat from core client for 30 sec - exiting 20:37:27 (7972): No heartbeat from core client for 30 sec - exiting 20:37:28 (7972): No heartbeat from core client for 30 sec - exiting 20:37:29 (7972): No heartbeat from core client for 30 sec - exiting 20:37:30 (7972): No heartbeat from core client for 30 sec - exiting 20:37:31 (7972): No heartbeat from core client for 30 sec - exiting 20:37:32 (7972): No heartbeat from core client for 30 sec - exiting 20:37:33 (7972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3992, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3992, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Skipping gmts_generator due to netcdf error 13 - Permission denied Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774C71B8 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o8ee_1900_40_008466009/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Jan 2014 14:16:03	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	1,036,800	3,096,273	2.9864
30 Dec 2013 21:17:16	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	1,010,880	3,006,821	2.9745
28 Dec 2013 16:27:38	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	984,960	2,918,621	2.9632
26 Dec 2013 22:04:18	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	959,040	2,831,717	2.9527
24 Dec 2013 21:07:39	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	933,120	2,747,963	2.9449
23 Dec 2013 02:14:29	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	907,200	2,667,050	2.9399
21 Dec 2013 06:25:49	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	881,280	2,578,704	2.9261
17 Dec 2013 22:23:32	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	855,360	2,493,716	2.9154
15 Dec 2013 04:00:20	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	829,440	2,406,105	2.9009
12 Dec 2013 05:24:32	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	803,520	2,321,212	2.8888
09 Dec 2013 22:12:36	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	777,600	2,232,060	2.8704
08 Dec 2013 04:32:59	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	751,680	2,146,064	2.8550
04 Dec 2013 19:38:02	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	725,760	2,056,690	2.8338
02 Dec 2013 03:37:37	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	699,840	1,970,322	2.8154
30 Nov 2013 05:24:04	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	673,920	1,878,952	2.7881
27 Nov 2013 06:00:11	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	648,000	1,793,106	2.7671
25 Nov 2013 22:37:50	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	622,080	1,717,067	2.7602
24 Nov 2013 14:58:05	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	596,160	1,652,547	2.7720
23 Nov 2013 19:23:18	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	570,240	1,612,322	2.8274
22 Nov 2013 07:05:30	1179495	16062329	hadcm3n_o8ee_1900_40_008466009_2	544,320	1,547,149	2.8424