climateprediction.net home page
Task 17229992

Task 17229992

Name hadcm3n_s0w8_1940_40_009094321_0
Workunit 9224657
Created 20 Oct 2014, 17:02:53 UTC
Sent 20 Oct 2014, 18:18:36 UTC
Report deadline 20 Jan 2015, 1:45:47 UTC
Received 22 Mar 2015, 22:40:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1313539
Run time 12 days 21 hours 32 min 3 sec
CPU time 10 days 14 hours 50 min 15 sec
Validate state Invalid
Credit 9,953.28
Device peak FLOPS 2.84 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr
<core_client_version>7.4.36</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
lfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel x86-64h Haswell (32-bit executable)
System version: Macintosh OS 10.10.3 build 14D87p
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1812e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(666,0xa0ea81d4) malloc: *** error for object 0x1812e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Tue Mar  3 13:48:21 2015

hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x821e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x821e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(545,0xa0ea81d4) malloc: *** error for object 0x280dc04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x100dc04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x100dc00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4805c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4805c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa0ea81d4) malloc: *** error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel x86-64h Haswell (32-bit executable)
System version: Macintosh OS 10.10.3 build 14D87p
Thu Mar  5 20:04:31 2015

17:26:16 (668): No heartbeat from core client for 30 sec - exiting
17:26:17 (668): No heartbeat from core client for 30 sec - exiting
17:26:18 (668): No heartbeat from core client for 30 sec - exiting
17:26:19 (668): No heartbeat from core client for 30 sec - exiting
16:23:33 (668): No heartbeat from core client for 30 sec - exiting
16:23:34 (668): No heartbeat from core client for 30 sec - exiting
16:23:35 (668): No heartbeat from core client for 30 sec - exiting
16:23:36 (668): No heartbeat from core client for 30 sec - exiting
16:23:37 (668): No heartbeat from core client for 30 sec - exiting
16:23:38 (668): No heartbeat from core client for 30 sec - exiting
16:23:39 (668): No heartbeat from core client for 30 sec - exiting
16:23:40 (668): No heartbeat from core client for 30 sec - exiting
16:23:41 (668): No heartbeat from core client for 30 sec - exiting
16:23:42 (668): No heartbeat from core client for 30 sec - exiting
16:23:43 (668): No heartbeat from core client for 30 sec - exiting
16:23:44 (668): No heartbeat from core client for 30 sec - exiting
16:23:45 (668): No heartbeat from core client for 30 sec - exiting
16:23:46 (668): No heartbeat from core client for 30 sec - exiting
16:23:47 (668): No heartbeat from core client for 30 sec - exiting
16:23:48 (668): No heartbeat from core client for 30 sec - exiting
16:23:49 (668): No heartbeat from core client for 30 sec - exiting
16:23:50 (668): No heartbeat from core client for 30 sec - exiting
16:23:51 (668): No heartbeat from core client for 30 sec - exiting
16:23:52 (668): No heartbeat from core client for 30 sec - exiting
16:23:53 (668): No heartbeat from core client for 30 sec - exiting
16:23:54 (668): No heartbeat from core client for 30 sec - exiting
16:23:55 (668): No heartbeat from core client for 30 sec - exiting
16:23:56 (668): No heartbeat from core client for 30 sec - exiting
16:23:57 (668): No heartbeat from core client for 30 sec - exiting
16:23:58 (668): No heartbeat from core client for 30 sec - exiting
16:23:59 (668): No heartbeat from core client for 30 sec - exiting
16:24:00 (668): No heartbeat from core client for 30 sec - exiting
16:24:01 (668): No heartbeat from core client for 30 sec - exiting
16:24:02 (668): No heartbeat from core client for 30 sec - exiting
16:24:03 (668): No heartbeat from core client for 30 sec - exiting
16:24:04 (668): No heartbeat from core client for 30 sec - exiting
16:24:05 (668): No heartbeat from core client for 30 sec - exiting
16:24:07 (668): No heartbeat from core client for 30 sec - exiting
16:24:08 (668): No heartbeat from core client for 30 sec - exiting
16:24:09 (668): No heartbeat from core client for 30 sec - exiting
16:24:10 (668): No heartbeat from core client for 30 sec - exiting
16:24:11 (668): No heartbeat from core client for 30 sec - exiting
16:24:12 (668): No heartbeat from core client for 30 sec - exiting
16:24:13 (668): No heartbeat from core client for 30 sec - exiting
16:24:14 (668): No heartbeat from core client for 30 sec - exiting
16:24:15 (668): No heartbeat from core client for 30 sec - exiting
16:24:16 (668): No heartbeat from core client for 30 sec - exiting
16:24:17 (668): No heartbeat from core client for 30 sec - exiting
16:24:18 (668): No heartbeat from core client for 30 sec - exiting
16:24:19 (668): No heartbeat from core client for 30 sec - exiting
16:24:20 (668): No heartbeat from core client for 30 sec - exiting
16:24:21 (668): No heartbeat from core client for 30 sec - exiting
16:24:22 (668): No heartbeat from core client for 30 sec - exiting
16:24:23 (668): No heartbeat from core client for 30 sec - exiting
16:24:24 (668): No heartbeat from core client for 30 sec - exiting
16:24:25 (668): No heartbeat from core client for 30 sec - exiting
16:24:26 (668): No heartbeat from core client for 30 sec - exiting
16:24:27 (668): No heartbeat from core client for 30 sec - exiting
16:24:28 (668): No heartbeat from core client for 30 sec - exiting
16:24:29 (668): No heartbeat from core client for 30 sec - exiting
16:24:30 (668): No heartbeat from core client for 30 sec - exiting
16:24:31 (668): No heartbeat from core client for 30 sec - exiting
16:24:32 (668): No heartbeat from core client for 30 sec - exiting
16:24:33 (668): No heartbeat from core client for 30 sec - exiting
16:24:34 (668): No heartbeat from core client for 30 sec - exiting
16:24:35 (668): No heartbeat from core client for 30 sec - exiting
16:24:36 (668): No heartbeat from core client for 30 sec - exiting
16:24:37 (668): No heartbeat from core client for 30 sec - exiting
16:24:38 (668): No heartbeat from core client for 30 sec - exiting
16:24:39 (668): No heartbeat from core client for 30 sec - exiting
16:24:40 (668): No heartbeat from core client for 30 sec - exiting
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x1015e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x1015e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(668,0xa114b1d4) malloc: *** error for object 0x809e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
17:20:55 (668): No heartbeat from core client for 30 sec - exiting
17:20:56 (668): No heartbeat from core client for 30 sec - exiting
17:20:57 (668): No heartbeat from core client for 30 sec - exiting
17:20:58 (668): No heartbeat from core client for 30 sec - exiting
17:20:59 (668): No heartbeat from core client for 30 sec - exiting
17:21:00 (668): No heartbeat from core client for 30 sec - exiting
17:21:01 (668): No heartbeat from core client for 30 sec - exiting
17:21:02 (668): No heartbeat from core client for 30 sec - exiting
17:21:03 (668): No heartbeat from core client for 30 sec - exiting
17:21:04 (668): No heartbeat from core client for 30 sec - exiting
17:21:05 (668): No heartbeat from core client for 30 sec - exiting
17:21:06 (668): No heartbeat from core client for 30 sec - exiting
17:21:07 (668): No heartbeat from core client for 30 sec - exiting
17:21:08 (668): No heartbeat from core client for 30 sec - exiting
17:21:09 (668): No heartbeat from core client for 30 sec - exiting
17:21:10 (668): No heartbeat from core client for 30 sec - exiting
17:21:12 (668): No heartbeat from core client for 30 sec - exiting
17:21:13 (668): No heartbeat from core client for 30 sec - exiting
17:21:14 (668): No heartbeat from core client for 30 sec - exiting
17:21:15 (668): No heartbeat from core client for 30 sec - exiting
17:21:16 (668): No heartbeat from core client for 30 sec - exiting
17:21:17 (668): No heartbeat from core client for 30 sec - exiting
17:21:18 (668): No heartbeat from core client for 30 sec - exiting
17:21:19 (668): No heartbeat from core client for 30 sec - exiting
17:21:20 (668): No heartbeat from core client for 30 sec - exiting
17:21:21 (668): No heartbeat from core client for 30 sec - exiting
17:21:22 (668): No heartbeat from core client for 30 sec - exiting
17:21:23 (668): No heartbeat from core client for 30 sec - exiting
17:21:24 (668): No heartbeat from core client for 30 sec - exiting
17:21:25 (668): No heartbeat from core client for 30 sec - exiting
17:21:26 (668): No heartbeat from core client for 30 sec - exiting
17:21:27 (668): No heartbeat from core client for 30 sec - exiting
08:40:32 (668): No heartbeat from core client for 30 sec - exiting
08:40:33 (668): No heartbeat from core client for 30 sec - exiting
08:40:34 (668): No heartbeat from core client for 30 sec - exiting
08:40:35 (668): No heartbeat from core client for 30 sec - exiting
08:40:36 (668): No heartbeat from core client for 30 sec - exiting
08:40:37 (668): No heartbeat from core client for 30 sec - exiting
08:40:38 (668): No heartbeat from core client for 30 sec - exiting
08:40:39 (668): No heartbeat from core client for 30 sec - exiting
08:40:40 (668): No heartbeat from core client for 30 sec - exiting
08:40:41 (668): No heartbeat from core client for 30 sec - exiting
08:40:42 (668): No heartbeat from core client for 30 sec - exiting
08:40:43 (668): No heartbeat from core client for 30 sec - exiting
08:40:44 (668): No heartbeat from core client for 30 sec - exiting
08:40:45 (668): No heartbeat from core client for 30 sec - exiting
08:40:46 (668): No heartbeat from core client for 30 sec - exiting
08:40:47 (668): No heartbeat from core client for 30 sec - exiting
08:40:48 (668): No heartbeat from core client for 30 sec - exiting
08:40:50 (668): No heartbeat from core client for 30 sec - exiting
08:40:51 (668): No heartbeat from core client for 30 sec - exiting
08:40:52 (668): No heartbeat from core client for 30 sec - exiting
08:40:53 (668): No heartbeat from core client for 30 sec - exiting
08:40:54 (668): No heartbeat from core client for 30 sec - exiting
08:40:55 (668): No heartbeat from core client for 30 sec - exiting
08:40:56 (668): No heartbeat from core client for 30 sec - exiting
08:40:57 (668): No heartbeat from core client for 30 sec - exiting
08:40:58 (668): No heartbeat from core client for 30 sec - exiting
08:40:59 (668): No heartbeat from core client for 30 sec - exiting
08:41:00 (668): No heartbeat from core client for 30 sec - exiting
08:41:01 (668): No heartbeat from core client for 30 sec - exiting
08:41:02 (668): No heartbeat from core client for 30 sec - exiting
08:41:03 (668): No heartbeat from core client for 30 sec - exiting
08:41:04 (668): No heartbeat from core client for 30 sec - exiting
08:41:05 (668): No heartbeat from core client for 30 sec - exiting
08:41:06 (668): No heartbeat from core client for 30 sec - exiting
08:41:07 (668): No heartbeat from core client for 30 sec - exiting
08:41:08 (668): No heartbeat from core client for 30 sec - exiting
08:41:09 (668): No heartbeat from core client for 30 sec - exiting
08:41:10 (668): No heartbeat from core client for 30 sec - exiting
08:41:11 (668): No heartbeat from core client for 30 sec - exiting
08:41:12 (668): No heartbeat from core client for 30 sec - exiting
08:41:13 (668): No heartbeat from core client for 30 sec - exiting
08:41:14 (668): No heartbeat from core client for 30 sec - exiting
08:41:16 (668): No heartbeat from core client for 30 sec - exiting
08:41:17 (668): No heartbeat from core client for 30 sec - exiting
08:41:18 (668): No heartbeat from core client for 30 sec - exiting
08:41:19 (668): No heartbeat from core client for 30 sec - exiting
08:41:20 (668): No heartbeat from core client for 30 sec - exiting
08:41:21 (668): No heartbeat from core client for 30 sec - exiting
08:41:22 (668): No heartbeat from core client for 30 sec - exiting
08:41:23 (668): No heartbeat from core client for 30 sec - exiting
08:41:24 (668): No heartbeat from core client for 30 sec - exiting
08:41:25 (668): No heartbeat from core client for 30 sec - exiting
08:41:26 (668): No heartbeat from core client for 30 sec - exiting
08:41:27 (668): No heartbeat from core client for 30 sec - exiting
08:41:28 (668): No heartbeat from core client for 30 sec - exiting
08:41:29 (668): No heartbeat from core client for 30 sec - exiting
08:41:30 (668): No heartbeat from core client for 30 sec - exiting
08:41:31 (668): No heartbeat from core client for 30 sec - exiting
08:41:32 (668): No heartbeat from core client for 30 sec - exiting
08:41:33 (668): No heartbeat from core client for 30 sec - exiting
08:41:34 (668): No heartbeat from core client for 30 sec - exiting
08:41:35 (668): No heartbeat from core client for 30 sec - exiting
08:41:36 (668): No heartbeat from core client for 30 sec - exiting
08:41:37 (668): No heartbeat from core client for 30 sec - exiting
08:41:39 (668): No heartbeat from core client for 30 sec - exiting
08:41:40 (668): No heartbeat from core client for 30 sec - exiting
08:41:41 (668): No heartbeat from core client for 30 sec - exiting
08:41:42 (668): No heartbeat from core client for 30 sec - exiting
08:41:43 (668): No heartbeat from core client for 30 sec - exiting
08:41:44 (668): No heartbeat from core client for 30 sec - exiting
08:41:45 (668): No heartbeat from core client for 30 sec - exiting
08:41:46 (668): No heartbeat from core client for 30 sec - exiting
08:41:47 (668): No heartbeat from core client for 30 sec - exiting
08:41:48 (668): No heartbeat from core client for 30 sec - exiting
08:41:49 (668): No heartbeat from core client for 30 sec - exiting
08:41:50 (668): No heartbeat from core client for 30 sec - exiting
08:41:51 (668): No heartbeat from core client for 30 sec - exiting
08:41:52 (668): No heartbeat from core client for 30 sec - exiting
08:41:53 (668): No heartbeat from core client for 30 sec - exiting
08:41:54 (668): No heartbeat from core client for 30 sec - exiting
08:41:55 (668): No heartbeat from core client for 30 sec - exiting
08:41:56 (668): No heartbeat from core client for 30 sec - exiting
08:41:57 (668): No heartbeat from core client for 30 sec - exiting
08:41:58 (668): No heartbeat from core client for 30 sec - exiting
08:41:59 (668): No heartbeat from core client for 30 sec - exiting
08:42:00 (668): No heartbeat from core client for 30 sec - exiting
08:42:01 (668): No heartbeat from core client for 30 sec - exiting
08:42:02 (668): No heartbeat from core client for 30 sec - exiting
08:42:03 (668): No heartbeat from core client for 30 sec - exiting
08:42:05 (668): No heartbeat from core client for 30 sec - exiting
08:42:06 (668): No heartbeat from core client for 30 sec - exiting
08:42:07 (668): No heartbeat from core client for 30 sec - exiting
08:42:08 (668): No heartbeat from core client for 30 sec - exiting
08:42:09 (668): No heartbeat from core client for 30 sec - exiting
08:42:10 (668): No heartbeat from core client for 30 sec - exiting
08:42:11 (668): No heartbeat from core client for 30 sec - exiting
08:42:12 (668): No heartbeat from core client for 30 sec - exiting
08:42:13 (668): No heartbeat from core client for 30 sec - exiting
08:42:14 (668): No heartbeat from core client for 30 sec - exiting
08:42:15 (668): No heartbeat from core client for 30 sec - exiting
08:42:16 (668): No heartbeat from core client for 30 sec - exiting
08:42:17 (668): No heartbeat from core client for 30 sec - exiting
08:42:18 (668): No heartbeat from core client for 30 sec - exiting
08:42:19 (668): No heartbeat from core client for 30 sec - exiting
08:42:20 (668): No heartbeat from core client for 30 sec - exiting
08:42:21 (668): No heartbeat from core client for 30 sec - exiting
08:42:22 (668): No heartbeat from core client for 30 sec - exiting
08:42:24 (668): No heartbeat from core client for 30 sec - exiting
08:42:25 (668): No heartbeat from core client for 30 sec - exiting
08:42:26 (668): No heartbeat from core client for 30 sec - exiting
08:42:27 (668): No heartbeat from core client for 30 sec - exiting
08:42:28 (668): No heartbeat from core client for 30 sec - exiting
08:42:29 (668): No heartbeat from core client for 30 sec - exiting
08:42:30 (668): No heartbeat from core client for 30 sec - exiting
08:42:31 (668): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x1005e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x1005e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x2031404: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x2031400: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(860,0xa06b61d4) malloc: *** error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel x86-64h Haswell (32-bit executable)
System version: Macintosh OS 10.10.3 build 14D105g
Tue Mar 17 08:48:30 2015

08:48:59 (860): No heartbeat from core client for 30 sec - exiting
08:49:00 (860): No heartbeat from core client for 30 sec - exiting
08:49:01 (860): No heartbeat from core client for 30 sec - exiting
08:49:02 (860): No heartbeat from core client for 30 sec - exiting
08:49:03 (860): No heartbeat from core client for 30 sec - exiting
08:49:04 (860): No heartbeat from core client for 30 sec - exiting
08:49:05 (860): No heartbeat from core client for 30 sec - exiting
08:49:06 (860): No heartbeat from core client for 30 sec - exiting
08:49:07 (860): No heartbeat from core client for 30 sec - exiting
08:49:08 (860): No heartbeat from core client for 30 sec - exiting
08:49:10 (860): No heartbeat from core client for 30 sec - exiting
08:49:11 (860): No heartbeat from core client for 30 sec - exiting
08:49:12 (860): No heartbeat from core client for 30 sec - exiting
08:49:13 (860): No heartbeat from core client for 30 sec - exiting
08:49:14 (860): No heartbeat from core client for 30 sec - exiting
08:49:15 (860): No heartbeat from core client for 30 sec - exiting
08:49:16 (860): No heartbeat from core client for 30 sec - exiting
08:49:17 (860): No heartbeat from core client for 30 sec - exiting
08:49:18 (860): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x281ac04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x281ac00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x401d200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2612,0xa05d81d4) malloc: *** error for object 0x1004e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel x86-64h Haswell (32-bit executable)
System version: Macintosh OS 10.10.3 build 14D105g
Thu Mar 19 09:53:33 2015

19:07:04 (2612): No heartbeat from core client for 30 sec - exiting
19:07:05 (2612): No heartbeat from core client for 30 sec - exiting
19:07:06 (2612): No heartbeat from core client for 30 sec - exiting
19:07:07 (2612): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x303c204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x303c200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(2362,0xa05d81d4) malloc: *** error for object 0x2005c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(713,0xa05d81d4) malloc: *** error for object 0x819e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel x86-64h Haswell (32-bit executable)
System version: Macintosh OS 10.10.3 build 14D105g
Sun Mar 22 18:39:06 2015

Thread 0 Crashed:

Thread 1:

atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386.
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd68 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -    0x93fff /Library/Application Support/BOINC Data/slots/0/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin
0x90008000 - 0x90008fff /usr/lib/system/liblaunch.dylib
0x9005d000 - 0x9005dfff /usr/lib/system/libunc.dylib
0x9034f000 - 0x9036efff /usr/lib/system/libsystem_kernel.dylib
0x907ba000 - 0x907c3fff /usr/lib/system/libcopyfile.dylib
0x90d9b000 - 0x90d9efff /usr/lib/system/libdyld.dylib
0x90d9f000 - 0x90da4fff /usr/lib/system/libcompiler_rt.dylib
0x90db3000 - 0x90db9fff /usr/lib/system/libsystem_platform.dylib
0x9183c000 - 0x91855fff /usr/lib/system/libsystem_malloc.dylib
0x91896000 - 0x9189efff /usr/lib/system/libsystem_pthread.dylib
0x91f07000 - 0x91f13fff /usr/lib/system/libcommonCrypto.dylib
0x925ac000 - 0x925c3fff /usr/lib/system/libsystem_asl.dylib
0x925c4000 - 0x92659fff /usr/lib/system/libsystem_c.dylib
0x931d5000 - 0x931defff /usr/lib/system/libsystem_notify.dylib
0x9346b000 - 0x93472fff /usr/lib/system/libunwind.dylib
0x956ca000 - 0x956f3fff /usr/lib/system/libsystem_info.dylib
0x95736000 - 0x9575cfff /usr/lib/libc++abi.dylib
0x9589a000 - 0x958edfff /usr/lib/libstdc++.6.dylib
0x96785000 - 0x967b8fff /usr/lib/system/libsystem_m.dylib
0x97c4e000 - 0x97c54fff /usr/lib/system/libsystem_networkextension.dylib
0x97d5a000 - 0x97d5afff /usr/lib/system/libkeymgr.dylib
0x97d5b000 - 0x97d6cfff /usr/lib/system/libsystem_coretls.dylib
0x986b1000 - 0x986b3fff /usr/lib/system/libsystem_coreservices.dylib
0x986b4000 - 0x986dbfff /usr/lib/system/libdispatch.dylib
0x98c97000 - 0x98c99fff /usr/lib/system/libsystem_configuration.dylib
0x98c9a000 - 0x98c9bfff /usr/lib/libSystem.B.dylib
0x98f8b000 - 0x98f8dfff /usr/lib/system/libquarantine.dylib
0x993ee000 - 0x993f2fff /usr/lib/system/libcache.dylib
0x997b8000 - 0x997bafff /usr/lib/system/libsystem_sandbox.dylib
0x99987000 - 0x999adfff /usr/lib/system/libxpc.dylib
0x99d35000 - 0x99d3bfff /usr/lib/system/libsystem_trace.dylib
0x99d3d000 - 0x99d3efff /usr/lib/system/libremovefile.dylib
0x99eee000 - 0x99efcfff /usr/lib/libz.1.dylib
0x9a9eb000 - 0x9aa28fff /usr/lib/system/libsystem_network.dylib
0x9aa29000 - 0x9aa2efff /usr/lib/system/libmacho.dylib
0x9aaf0000 - 0x9aaf2fff /usr/lib/system/libsystem_secinit.dylib
0x9b222000 - 0x9b22afff /usr/lib/system/libsystem_dnssd.dylib
0x9b22b000 - 0x9b22cfff /usr/lib/system/libsystem_blocks.dylib
0x9bcc9000 - 0x9bd32fff /usr/lib/system/libcorecrypto.dylib


Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
03 Mar 2015 18:51:53 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 829,440 917,415 1.1061
28 Feb 2015 21:13:09 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 777,600 877,269 1.1282
26 Feb 2015 18:11:48 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 751,680 859,781 1.1438
17 Feb 2015 22:57:32 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 725,760 842,848 1.1613
16 Feb 2015 02:17:16 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 699,840 826,253 1.1806
12 Feb 2015 07:46:35 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 673,920 804,517 1.1938
09 Feb 2015 00:47:43 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 648,000 785,683 1.2125
07 Feb 2015 00:24:18 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 622,080 767,998 1.2346
04 Feb 2015 03:43:27 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 596,160 751,143 1.2600
01 Feb 2015 01:04:37 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 570,240 735,670 1.2901
31 Jan 2015 02:44:16 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 544,320 719,668 1.3221
30 Jan 2015 08:33:59 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 518,400 697,021 1.3446
29 Jan 2015 01:29:02 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 492,480 681,317 1.3834
22 Jan 2015 00:38:21 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 466,560 670,404 1.4369
18 Jan 2015 02:21:18 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 440,640 633,770 1.4383
12 Jan 2015 18:35:23 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 414,720 596,916 1.4393
04 Jan 2015 23:00:51 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 388,800 559,855 1.4400
04 Jan 2015 06:27:50 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 362,880 523,517 1.4427
01 Jan 2015 21:19:37 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 336,960 486,448 1.4436
28 Dec 2014 00:06:13 1313539 17229992 hadcm3n_s0w8_1940_40_009094321_0 311,040 448,751 1.4427


©2024 climateprediction.net