Task 15071465

Name	hadcm3n_006o_1980_40_008000673_2
Workunit	8155787
Created	6 Aug 2012, 13:52:49 UTC
Sent	6 Aug 2012, 13:53:10 UTC
Report deadline	5 Nov 2012, 21:20:21 UTC
Received	22 Oct 2012, 7:46:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1179185
Run time	28 days 16 hours 34 min 56 sec
CPU time	28 days 16 hours 34 min 56 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	1.43 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09b699d8 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 14:34:19 (1273): No heartbeat from core client for 30 sec - exiting 14:35:29 (1273): No heartbeat from core client for 30 sec - exiting 14:35:30 (1273): No heartbeat from core client for 30 sec - exiting 14:35:31 (1273): No heartbeat from core client for 30 sec - exiting 14:35:32 (1273): No heartbeat from core client for 30 sec - exiting 14:35:33 (1273): No heartbeat from core client for 30 sec - exiting 14:35:34 (1273): No heartbeat from core client for 30 sec - exiting 14:35:35 (1273): No heartbeat from core client for 30 sec - exiting 14:35:36 (1273): No heartbeat from core client for 30 sec - exiting 14:35:37 (1273): No heartbeat from core client for 30 sec - exiting 14:35:38 (1273): No heartbeat from core client for 30 sec - exiting 14:35:39 (1273): No heartbeat from core client for 30 sec - exiting 14:35:40 (1273): No heartbeat from core client for 30 sec - exiting 14:35:41 (1273): No heartbeat from core client for 30 sec - exiting 14:35:42 (1273): No heartbeat from core client for 30 sec - exiting 14:35:43 (1273): No heartbeat from core client for 30 sec - exiting 14:35:44 (1273): No heartbeat from core client for 30 sec - exiting 14:35:45 (1273): No heartbeat from core client for 30 sec - exiting 14:35:46 (1273): No heartbeat from core client for 30 sec - exiting 14:35:47 (1273): No heartbeat from core client for 30 sec - exiting 14:35:48 (1273): No heartbeat from core client for 30 sec - exiting 14:35:49 (1273): No heartbeat from core client for 30 sec - exiting 14:35:50 (1273): No heartbeat from core client for 30 sec - exiting 14:35:51 (1273): No heartbeat from core client for 30 sec - exiting 14:36:13 (1273): No heartbeat from core client for 30 sec - exiting 14:36:14 (1273): No heartbeat from core client for 30 sec - exiting 14:36:15 (1273): No heartbeat from core client for 30 sec - exiting 14:36:16 (1273): No heartbeat from core client for 30 sec - exiting 14:36:17 (1273): No heartbeat from core client for 30 sec - exiting 14:36:18 (1273): No heartbeat from core client for 30 sec - exiting 14:36:19 (1273): No heartbeat from core client for 30 sec - exiting 14:36:20 (1273): No heartbeat from core client for 30 sec - exiting 14:36:21 (1273): No heartbeat from core client for 30 sec - exiting 14:36:22 (1273): No heartbeat from core client for 30 sec - exiting 14:36:23 (1273): No heartbeat from core client for 30 sec - exiting 14:36:24 (1273): No heartbeat from core client for 30 sec - exiting 14:36:25 (1273): No heartbeat from core client for 30 sec - exiting 14:36:26 (1273): No heartbeat from core client for 30 sec - exiting 14:36:27 (1273): No heartbeat from core client for 30 sec - exiting 14:36:28 (1273): No heartbeat from core client for 30 sec - exiting 14:36:29 (1273): No heartbeat from core client for 30 sec - exiting 14:36:30 (1273): No heartbeat from core client for 30 sec - exiting 14:36:31 (1273): No heartbeat from core client for 30 sec - exiting 14:36:32 (1273): No heartbeat from core client for 30 sec - exiting 14:36:33 (1273): No heartbeat from core client for 30 sec - exiting 14:36:34 (1273): No heartbeat from core client for 30 sec - exiting 14:36:35 (1273): No heartbeat from core client for 30 sec - exiting 14:36:36 (1273): No heartbeat from core client for 30 sec - exiting 14:36:37 (1273): No heartbeat from core client for 30 sec - exiting 14:36:38 (1273): No heartbeat from core client for 30 sec - exiting 14:36:39 (1273): No heartbeat from core client for 30 sec - exiting 14:36:40 (1273): No heartbeat from core client for 30 sec - exiting 14:36:41 (1273): No heartbeat from core client for 30 sec - exiting 14:36:42 (1273): No heartbeat from core client for 30 sec - exiting 14:36:43 (1273): No heartbeat from core client for 30 sec - exiting 14:36:44 (1273): No heartbeat from core client for 30 sec - exiting 14:36:45 (1273): No heartbeat from core client for 30 sec - exiting 14:36:46 (1273): No heartbeat from core client for 30 sec - exiting 14:36:47 (1273): No heartbeat from core client for 30 sec - exiting 14:36:48 (1273): No heartbeat from core client for 30 sec - exiting 14:36:49 (1273): No heartbeat from core client for 30 sec - exiting 14:36:50 (1273): No heartbeat from core client for 30 sec - exiting 14:36:51 (1273): No heartbeat from core client for 30 sec - exiting 14:36:52 (1273): No heartbeat from core client for 30 sec - exiting 14:36:53 (1273): No heartbeat from core client for 30 sec - exiting 14:36:54 (1273): No heartbeat from core client for 30 sec - exiting 14:36:55 (1273): No heartbeat from core client for 30 sec - exiting 14:36:56 (1273): No heartbeat from core client for 30 sec - exiting 14:36:57 (1273): No heartbeat from core client for 30 sec - exiting 14:36:58 (1273): No heartbeat from core client for 30 sec - exiting 14:36:59 (1273): No heartbeat from core client for 30 sec - exiting 14:37:00 (1273): No heartbeat from core client for 30 sec - exiting 14:37:01 (1273): No heartbeat from core client for 30 sec - exiting 14:37:02 (1273): No heartbeat from core client for 30 sec - exiting 14:37:03 (1273): No heartbeat from core client for 30 sec - exiting 14:37:04 (1273): No heartbeat from core client for 30 sec - exiting 14:37:05 (1273): No heartbeat from core client for 30 sec - exiting 14:37:06 (1273): No heartbeat from core client for 30 sec - exiting 14:37:07 (1273): No heartbeat from core client for 30 sec - exiting 14:37:08 (1273): No heartbeat from core client for 30 sec - exiting 14:37:09 (1273): No heartbeat from core client for 30 sec - exiting 14:37:10 (1273): No heartbeat from core client for 30 sec - exiting 14:37:11 (1273): No heartbeat from core client for 30 sec - exiting 14:37:12 (1273): No heartbeat from core client for 30 sec - exiting 14:37:13 (1273): No heartbeat from core client for 30 sec - exiting 14:37:14 (1273): No heartbeat from core client for 30 sec - exiting 14:37:15 (1273): No heartbeat from core client for 30 sec - exiting 14:37:16 (1273): No heartbeat from core client for 30 sec - exiting 14:37:17 (1273): No heartbeat from core client for 30 sec - exiting 14:37:18 (1273): No heartbeat from core client for 30 sec - exiting 14:37:19 (1273): No heartbeat from core client for 30 sec - exiting 14:37:20 (1273): No heartbeat from core client for 30 sec - exiting 14:37:21 (1273): No heartbeat from core client for 30 sec - exiting 14:37:22 (1273): No heartbeat from core client for 30 sec - exiting 14:37:23 (1273): No heartbeat from core client for 30 sec - exiting 14:37:24 (1273): No heartbeat from core client for 30 sec - exiting 14:37:25 (1273): No heartbeat from core client for 30 sec - exiting 14:37:26 (1273): No heartbeat from core client for 30 sec - exiting 14:37:27 (1273): No heartbeat from core client for 30 sec - exiting 14:37:28 (1273): No heartbeat from core client for 30 sec - exiting 14:37:29 (1273): No heartbeat from core client for 30 sec - exiting 14:37:30 (1273): No heartbeat from core client for 30 sec - exiting 14:37:31 (1273): No heartbeat from core client for 30 sec - exiting 14:37:32 (1273): No heartbeat from core client for 30 sec - exiting 14:37:33 (1273): No heartbeat from core client for 30 sec - exiting 14:37:34 (1273): No heartbeat from core client for 30 sec - exiting 14:37:35 (1273): No heartbeat from core client for 30 sec - exiting 14:37:36 (1273): No heartbeat from core client for 30 sec - exiting 14:37:37 (1273): No heartbeat from core client for 30 sec - exiting 14:37:38 (1273): No heartbeat from core client for 30 sec - exiting 14:37:39 (1273): No heartbeat from core client for 30 sec - exiting 14:37:40 (1273): No heartbeat from core client for 30 sec - exiting 14:37:41 (1273): No heartbeat from core client for 30 sec - exiting 14:37:43 (1273): No heartbeat from core client for 30 sec - exiting 14:37:44 (1273): No heartbeat from core client for 30 sec - exiting 14:37:45 (1273): No heartbeat from core client for 30 sec - exiting 14:37:46 (1273): No heartbeat from core client for 30 sec - exiting 14:37:47 (1273): No heartbeat from core client for 30 sec - exiting 14:37:48 (1273): No heartbeat from core client for 30 sec - exiting 14:37:49 (1273): No heartbeat from core client for 30 sec - exiting 14:37:50 (1273): No heartbeat from core client for 30 sec - exiting 14:37:51 (1273): No heartbeat from core client for 30 sec - exiting 14:37:52 (1273): No heartbeat from core client for 30 sec - exiting 14:37:53 (1273): No heartbeat from core client for 30 sec - exiting 14:37:54 (1273): No heartbeat from core client for 30 sec - exiting 14:37:55 (1273): No heartbeat from core client for 30 sec - exiting 14:37:56 (1273): No heartbeat from core client for 30 sec - exiting 14:37:57 (1273): No heartbeat from core client for 30 sec - exiting 14:37:58 (1273): No heartbeat from core client for 30 sec - exiting 14:38:02 (1273): No heartbeat from core client for 30 sec - exiting 14:38:03 (1273): No heartbeat from core client for 30 sec - exiting 14:38:04 (1273): No heartbeat from core client for 30 sec - exiting 14:38:05 (1273): No heartbeat from core client for 30 sec - exiting 14:38:06 (1273): No heartbeat from core client for 30 sec - exiting 14:38:07 (1273): No heartbeat from core client for 30 sec - exiting 14:38:08 (1273): No heartbeat from core client for 30 sec - exiting 14:38:09 (1273): No heartbeat from core client for 30 sec - exiting 14:38:10 (1273): No heartbeat from core client for 30 sec - exiting 14:38:11 (1273): No heartbeat from core client for 30 sec - exiting 14:38:12 (1273): No heartbeat from core client for 30 sec - exiting 14:38:13 (1273): No heartbeat from core client for 30 sec - exiting 14:38:14 (1273): No heartbeat from core client for 30 sec - exiting 14:38:15 (1273): No heartbeat from core client for 30 sec - exiting 14:38:16 (1273): No heartbeat from core client for 30 sec - exiting 14:38:17 (1273): No heartbeat from core client for 30 sec - exiting 14:38:18 (1273): No heartbeat from core client for 30 sec - exiting 14:38:19 (1273): No heartbeat from core client for 30 sec - exiting 14:38:20 (1273): No heartbeat from core client for 30 sec - exiting 14:38:21 (1273): No heartbeat from core client for 30 sec - exiting 14:38:22 (1273): No heartbeat from core client for 30 sec - exiting 14:38:23 (1273): No heartbeat from core client for 30 sec - exiting 14:38:24 (1273): No heartbeat from core client for 30 sec - exiting 14:38:25 (1273): No heartbeat from core client for 30 sec - exiting 14:38:26 (1273): No heartbeat from core client for 30 sec - exiting 14:38:27 (1273): No heartbeat from core client for 30 sec - exiting 14:38:28 (1273): No heartbeat from core client for 30 sec - exiting 14:38:29 (1273): No heartbeat from core client for 30 sec - exiting 14:38:30 (1273): No heartbeat from core client for 30 sec - exiting 14:38:31 (1273): No heartbeat from core client for 30 sec - exiting 14:38:32 (1273): No heartbeat from core client for 30 sec - exiting 14:38:33 (1273): No heartbeat from core client for 30 sec - exiting 14:38:34 (1273): No heartbeat from core client for 30 sec - exiting 14:38:35 (1273): No heartbeat from core client for 30 sec - exiting 14:38:36 (1273): No heartbeat from core client for 30 sec - exiting 14:38:37 (1273): No heartbeat from core client for 30 sec - exiting 14:38:38 (1273): No heartbeat from core client for 30 sec - exiting 14:38:39 (1273): No heartbeat from core client for 30 sec - exiting 14:38:40 (1273): No heartbeat from core client for 30 sec - exiting 14:38:41 (1273): No heartbeat from core client for 30 sec - exiting 14:38:42 (1273): No heartbeat from core client for 30 sec - exiting 14:38:43 (1273): No heartbeat from core client for 30 sec - exiting 14:38:44 (1273): No heartbeat from core client for 30 sec - exiting 14:38:45 (1273): No heartbeat from core client for 30 sec - exiting 14:38:46 (1273): No heartbeat from core client for 30 sec - exiting 14:38:47 (1273): No heartbeat from core client for 30 sec - exiting 14:38:48 (1273): No heartbeat from core client for 30 sec - exiting 14:38:49 (1273): No heartbeat from core client for 30 sec - exiting 14:38:50 (1273): No heartbeat from core client for 30 sec - exiting 14:38:51 (1273): No heartbeat from core client for 30 sec - exiting 14:38:52 (1273): No heartbeat from core client for 30 sec - exiting 14:38:53 (1273): No heartbeat from core client for 30 sec - exiting 14:38:54 (1273): No heartbeat from core client for 30 sec - exiting 14:38:55 (1273): No heartbeat from core client for 30 sec - exiting 14:38:56 (1273): No heartbeat from core client for 30 sec - exiting 14:38:57 (1273): No heartbeat from core client for 30 sec - exiting 14:38:58 (1273): No heartbeat from core client for 30 sec - exiting 14:38:59 (1273): No heartbeat from core client for 30 sec - exiting 14:39:00 (1273): No heartbeat from core client for 30 sec - exiting 14:39:01 (1273): No heartbeat from core client for 30 sec - exiting 14:39:02 (1273): No heartbeat from core client for 30 sec - exiting 14:39:03 (1273): No heartbeat from core client for 30 sec - exiting 14:39:04 (1273): No heartbeat from core client for 30 sec - exiting 14:39:05 (1273): No heartbeat from core client for 30 sec - exiting 14:39:06 (1273): No heartbeat from core client for 30 sec - exiting 14:39:07 (1273): No heartbeat from core client for 30 sec - exiting 14:39:08 (1273): No heartbeat from core client for 30 sec - exiting 14:39:09 (1273): No heartbeat from core client for 30 sec - exiting 14:39:10 (1273): No heartbeat from core client for 30 sec - exiting 14:39:11 (1273): No heartbeat from core client for 30 sec - exiting 14:39:12 (1273): No heartbeat from core client for 30 sec - exiting 14:39:13 (1273): No heartbeat from core client for 30 sec - exiting 14:39:14 (1273): No heartbeat from core client for 30 sec - exiting 14:39:15 (1273): No heartbeat from core client for 30 sec - exiting 14:39:16 (1273): No heartbeat from core client for 30 sec - exiting 14:39:20 (1273): No heartbeat from core client for 30 sec - exiting 14:39:21 (1273): No heartbeat from core client for 30 sec - exiting 14:39:22 (1273): No heartbeat from core client for 30 sec - exiting 14:39:37 (1273): No heartbeat from core client for 30 sec - exiting 14:39:38 (1273): No heartbeat from core client for 30 sec - exiting 14:39:39 (1273): No heartbeat from core client for 30 sec - exiting 14:39:40 (1273): No heartbeat from core client for 30 sec - exiting 14:39:41 (1273): No heartbeat from core client for 30 sec - exiting 14:39:42 (1273): No heartbeat from core client for 30 sec - exiting 14:39:43 (1273): No heartbeat from core client for 30 sec - exiting 14:39:44 (1273): No heartbeat from core client for 30 sec - exiting 14:39:45 (1273): No heartbeat from core client for 30 sec - exiting 14:39:46 (1273): No heartbeat from core client for 30 sec - exiting 14:39:47 (1273): No heartbeat from core client for 30 sec - exiting 14:39:48 (1273): No heartbeat from core client for 30 sec - exiting 14:39:49 (1273): No heartbeat from core client for 30 sec - exiting 14:39:50 (1273): No heartbeat from core client for 30 sec - exiting 14:39:51 (1273): No heartbeat from core client for 30 sec - exiting 14:39:52 (1273): No heartbeat from core client for 30 sec - exiting 14:39:53 (1273): No heartbeat from core client for 30 sec - exiting 14:39:54 (1273): No heartbeat from core client for 30 sec - exiting 14:39:55 (1273): No heartbeat from core client for 30 sec - exiting 14:39:56 (1273): No heartbeat from core client for 30 sec - exiting 14:39:57 (1273): No heartbeat from core client for 30 sec - exiting 14:39:58 (1273): No heartbeat from core client for 30 sec - exiting 14:39:59 (1273): No heartbeat from core client for 30 sec - exiting 14:40:00 (1273): No heartbeat from core client for 30 sec - exiting 14:40:01 (1273): No heartbeat from core client for 30 sec - exiting 14:40:02 (1273): No heartbeat from core client for 30 sec - exiting 14:40:03 (1273): No heartbeat from core client for 30 sec - exiting 14:40:04 (1273): No heartbeat from core client for 30 sec - exiting 14:40:05 (1273): No heartbeat from core client for 30 sec - exiting 14:40:06 (1273): No heartbeat from core client for 30 sec - exiting 14:40:07 (1273): No heartbeat from core client for 30 sec - exiting 14:40:08 (1273): No heartbeat from core client for 30 sec - exiting 14:40:09 (1273): No heartbeat from core client for 30 sec - exiting 14:40:20 (1273): No heartbeat from core client for 30 sec - exiting 14:40:21 (1273): No heartbeat from core client for 30 sec - exiting 14:40:22 (1273): No heartbeat from core client for 30 sec - exiting 14:40:23 (1273): No heartbeat from core client for 30 sec - exiting 14:40:24 (1273): No heartbeat from core client for 30 sec - exiting 14:40:25 (1273): No heartbeat from core client for 30 sec - exiting 14:40:26 (1273): No heartbeat from core client for 30 sec - exiting 14:40:27 (1273): No heartbeat from core client for 30 sec - exiting 14:40:28 (1273): No heartbeat from core client for 30 sec - exiting 14:40:29 (1273): No heartbeat from core client for 30 sec - exiting 14:40:30 (1273): No heartbeat from core client for 30 sec - exiting 14:40:31 (1273): No heartbeat from core client for 30 sec - exiting 14:40:32 (1273): No heartbeat from core client for 30 sec - exiting 14:40:33 (1273): No heartbeat from core client for 30 sec - exiting 14:40:34 (1273): No heartbeat from core client for 30 sec - exiting 14:40:35 (1273): No heartbeat from core client for 30 sec - exiting 14:40:36 (1273): No heartbeat from core client for 30 sec - exiting 14:40:37 (1273): No heartbeat from core client for 30 sec - exiting 14:40:38 (1273): No heartbeat from core client for 30 sec - exiting 14:40:39 (1273): No heartbeat from core client for 30 sec - exiting 14:40:40 (1273): No heartbeat from core client for 30 sec - exiting 14:40:41 (1273): No heartbeat from core client for 30 sec - exiting 14:40:42 (1273): No heartbeat from core client for 30 sec - exiting 14:40:43 (1273): No heartbeat from core client for 30 sec - exiting 14:40:47 (1273): No heartbeat from core client for 30 sec - exiting 14:40:48 (1273): No heartbeat from core client for 30 sec - exiting 14:40:49 (1273): No heartbeat from core client for 30 sec - exiting 14:40:50 (1273): No heartbeat from core client for 30 sec - exiting 14:40:51 (1273): No heartbeat from core client for 30 sec - exiting 14:40:52 (1273): No heartbeat from core client for 30 sec - exiting 14:40:53 (1273): No heartbeat from core client for 30 sec - exiting 14:40:54 (1273): No heartbeat from core client for 30 sec - exiting 14:40:55 (1273): No heartbeat from core client for 30 sec - exiting 14:40:56 (1273): No heartbeat from core client for 30 sec - exiting 14:40:57 (1273): No heartbeat from core client for 30 sec - exiting 14:40:58 (1273): No heartbeat from core client for 30 sec - exiting 14:40:59 (1273): No heartbeat from core client for 30 sec - exiting 14:41:00 (1273): No heartbeat from core client for 30 sec - exiting 14:41:01 (1273): No heartbeat from core client for 30 sec - exiting 14:41:02 (1273): No heartbeat from core client for 30 sec - exiting 14:41:03 (1273): No heartbeat from core client for 30 sec - exiting 14:41:04 (1273): No heartbeat from core client for 30 sec - exiting 14:41:05 (1273): No heartbeat from core client for 30 sec - exiting 14:41:06 (1273): No heartbeat from core client for 30 sec - exiting 14:41:07 (1273): No heartbeat from core client for 30 sec - exiting 14:41:08 (1273): No heartbeat from core client for 30 sec - exiting 14:41:15 (1273): No heartbeat from core client for 30 sec - exiting 14:41:26 (1273): No heartbeat from core client for 30 sec - exiting 14:41:27 (1273): No heartbeat from core client for 30 sec - exiting 14:41:28 (1273): No heartbeat from core client for 30 sec - exiting 14:41:29 (1273): No heartbeat from core client for 30 sec - exiting 14:41:30 (1273): No heartbeat from core client for 30 sec - exiting 14:41:34 (1273): No heartbeat from core client for 30 sec - exiting 14:41:35 (1273): No heartbeat from core client for 30 sec - exiting 14:41:36 (1273): No heartbeat from core client for 30 sec - exiting 14:41:37 (1273): No heartbeat from core client for 30 sec - exiting 14:41:38 (1273): No heartbeat from core client for 30 sec - exiting 14:41:39 (1273): No heartbeat from core client for 30 sec - exiting 14:41:40 (1273): No heartbeat from core client for 30 sec - exiting 14:41:41 (1273): No heartbeat from core client for 30 sec - exiting 14:41:42 (1273): No heartbeat from core client for 30 sec - exiting 14:41:43 (1273): No heartbeat from core client for 30 sec - exiting 14:41:44 (1273): No heartbeat from core client for 30 sec - exiting 14:41:45 (1273): No heartbeat from core client for 30 sec - exiting 14:41:46 (1273): No heartbeat from core client for 30 sec - exiting 14:41:47 (1273): No heartbeat from core client for 30 sec - exiting 14:41:48 (1273): No heartbeat from core client for 30 sec - exiting 14:41:49 (1273): No heartbeat from core client for 30 sec - exiting 14:41:50 (1273): No heartbeat from core client for 30 sec - exiting 14:41:51 (1273): No heartbeat from core client for 30 sec - exiting 14:41:52 (1273): No heartbeat from core client for 30 sec - exiting 14:41:54 (1273): No heartbeat from core client for 30 sec - exiting 14:41:55 (1273): No heartbeat from core client for 30 sec - exiting 14:41:56 (1273): No heartbeat from core client for 30 sec - exiting 14:41:57 (1273): No heartbeat from core client for 30 sec - exiting 14:41:58 (1273): No heartbeat from core client for 30 sec - exiting 14:41:59 (1273): No heartbeat from core client for 30 sec - exiting 14:42:00 (1273): No heartbeat from core client for 30 sec - exiting 14:42:01 (1273): No heartbeat from core client for 30 sec - exiting 14:42:02 (1273): No heartbeat from core client for 30 sec - exiting 14:42:03 (1273): No heartbeat from core client for 30 sec - exiting 14:42:04 (1273): No heartbeat from core client for 30 sec - exiting 14:42:05 (1273): No heartbeat from core client for 30 sec - exiting 14:42:06 (1273): No heartbeat from core client for 30 sec - exiting 14:42:07 (1273): No heartbeat from core client for 30 sec - exiting 14:42:08 (1273): No heartbeat from core client for 30 sec - exiting 14:42:09 (1273): No heartbeat from core client for 30 sec - exiting 14:42:10 (1273): No heartbeat from core client for 30 sec - exiting 14:42:11 (1273): No heartbeat from core client for 30 sec - exiting 14:42:12 (1273): No heartbeat from core client for 30 sec - exiting 14:42:13 (1273): No heartbeat from core client for 30 sec - exiting 14:42:14 (1273): No heartbeat from core client for 30 sec - exiting 14:42:15 (1273): No heartbeat from core client for 30 sec - exiting 14:42:16 (1273): No heartbeat from core client for 30 sec - exiting 14:42:17 (1273): No heartbeat from core client for 30 sec - exiting 14:42:18 (1273): No heartbeat from core client for 30 sec - exiting 14:42:49 (1273): No heartbeat from core client for 30 sec - exiting 14:42:50 (1273): No heartbeat from core client for 30 sec - exiting 14:42:51 (1273): No heartbeat from core client for 30 sec - exiting 14:42:52 (1273): No heartbeat from core client for 30 sec - exiting 14:42:53 (1273): No heartbeat from core client for 30 sec - exiting 14:42:54 (1273): No heartbeat from core client for 30 sec - exiting 14:42:55 (1273): No heartbeat from core client for 30 sec - exiting 14:42:56 (1273): No heartbeat from core client for 30 sec - exiting 14:42:57 (1273): No heartbeat from core client for 30 sec - exiting 12:27:23 (1273): No heartbeat from core client for 30 sec - exiting 12:31:21 (1273): No heartbeat from core client for 30 sec - exiting 12:31:22 (1273): No heartbeat from core client for 30 sec - exiting 12:31:23 (1273): No heartbeat from core client for 30 sec - exiting 12:31:24 (1273): No heartbeat from core client for 30 sec - exiting 12:31:25 (1273): No heartbeat from core client for 30 sec - exiting 12:31:26 (1273): No heartbeat from core client for 30 sec - exiting 12:31:27 (1273): No heartbeat from core client for 30 sec - exiting 12:31:28 (1273): No heartbeat from core client for 30 sec - exiting 12:31:29 (1273): No heartbeat from core client for 30 sec - exiting 12:31:30 (1273): No heartbeat from core client for 30 sec - exiting 12:31:31 (1273): No heartbeat from core client for 30 sec - exiting 12:31:32 (1273): No heartbeat from core client for 30 sec - exiting 12:31:33 (1273): No heartbeat from core client for 30 sec - exiting 12:31:34 (1273): No heartbeat from core client for 30 sec - exiting 12:31:35 (1273): No heartbeat from core client for 30 sec - exiting 12:31:36 (1273): No heartbeat from core client for 30 sec - exiting 12:31:37 (1273): No heartbeat from core client for 30 sec - exiting 12:31:38 (1273): No heartbeat from core client for 30 sec - exiting 12:31:39 (1273): No heartbeat from core client for 30 sec - exiting 12:31:40 (1273): No heartbeat from core client for 30 sec - exiting 12:31:41 (1273): No heartbeat from core client for 30 sec - exiting 12:31:42 (1273): No heartbeat from core client for 30 sec - exiting 12:31:43 (1273): No heartbeat from core client for 30 sec - exiting 12:31:44 (1273): No heartbeat from core client for 30 sec - exiting 12:31:45 (1273): No heartbeat from core client for 30 sec - exiting 12:31:46 (1273): No heartbeat from core client for 30 sec - exiting 12:31:47 (1273): No heartbeat from core client for 30 sec - exiting 12:31:48 (1273): No heartbeat from core client for 30 sec - exiting 12:31:49 (1273): No heartbeat from core client for 30 sec - exiting 12:31:50 (1273): No heartbeat from core client for 30 sec - exiting 12:31:51 (1273): No heartbeat from core client for 30 sec - exiting 12:31:52 (1273): No heartbeat from core client for 30 sec - exiting 12:31:53 (1273): No heartbeat from core client for 30 sec - exiting 12:31:54 (1273): No heartbeat from core client for 30 sec - exiting 12:31:55 (1273): No heartbeat from core client for 30 sec - exiting 12:31:56 (1273): No heartbeat from core client for 30 sec - exiting 12:31:57 (1273): No heartbeat from core client for 30 sec - exiting 12:31:58 (1273): No heartbeat from core client for 30 sec - exiting 12:31:59 (1273): No heartbeat from core client for 30 sec - exiting 12:32:00 (1273): No heartbeat from core client for 30 sec - exiting 12:32:02 (1273): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x083d2318 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09fd2318 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 00:26:18 (1183): No heartbeat from core client for 30 sec - exiting 22:57:05 (1183): No heartbeat from core client for 30 sec - exiting 22:57:06 (1183): No heartbeat from core client for 30 sec - exiting 22:57:07 (1183): No heartbeat from core client for 30 sec - exiting 22:57:08 (1183): No heartbeat from core client for 30 sec - exiting 22:57:09 (1183): No heartbeat from core client for 30 sec - exiting 22:57:10 (1183): No heartbeat from core client for 30 sec - exiting 22:57:11 (1183): No heartbeat from core client for 30 sec - exiting 22:57:12 (1183): No heartbeat from core client for 30 sec - exiting 22:57:13 (1183): No heartbeat from core client for 30 sec - exiting 22:57:14 (1183): No heartbeat from core client for 30 sec - exiting 22:57:15 (1183): No heartbeat from core client for 30 sec - exiting 22:58:16 (1183): No heartbeat from core client for 30 sec - exiting 22:58:17 (1183): No heartbeat from core client for 30 sec - exiting 22:58:18 (1183): No heartbeat from core client for 30 sec - exiting 22:58:19 (1183): No heartbeat from core client for 30 sec - exiting 22:58:20 (1183): No heartbeat from core client for 30 sec - exiting 22:58:21 (1183): No heartbeat from core client for 30 sec - exiting 22:58:22 (1183): No heartbeat from core client for 30 sec - exiting 22:58:23 (1183): No heartbeat from core client for 30 sec - exiting 22:58:24 (1183): No heartbeat from core client for 30 sec - exiting 22:58:25 (1183): No heartbeat from core client for 30 sec - exiting 22:58:26 (1183): No heartbeat from core client for 30 sec - exiting 03:45:07 (1183): No heartbeat from core client for 30 sec - exiting 03:45:52 (1183): No heartbeat from core client for 30 sec - exiting 03:45:53 (1183): No heartbeat from core client for 30 sec - exiting 03:45:54 (1183): No heartbeat from core client for 30 sec - exiting 03:45:55 (1183): No heartbeat from core client for 30 sec - exiting 03:45:56 (1183): No heartbeat from core client for 30 sec - exiting 03:45:57 (1183): No heartbeat from core client for 30 sec - exiting 03:45:58 (1183): No heartbeat from core client for 30 sec - exiting 03:45:59 (1183): No heartbeat from core client for 30 sec - exiting 03:46:00 (1183): No heartbeat from core client for 30 sec - exiting 03:46:01 (1183): No heartbeat from core client for 30 sec - exiting 03:46:02 (1183): No heartbeat from core client for 30 sec - exiting 03:46:03 (1183): No heartbeat from core client for 30 sec - exiting 03:46:04 (1183): No heartbeat from core client for 30 sec - exiting 03:46:05 (1183): No heartbeat from core client for 30 sec - exiting 03:46:06 (1183): No heartbeat from core client for 30 sec - exiting 03:46:07 (1183): No heartbeat from core client for 30 sec - exiting 03:46:08 (1183): No heartbeat from core client for 30 sec - exiting 03:46:17 (1183): No heartbeat from core client for 30 sec - exiting 03:46:40 (1183): No heartbeat from core client for 30 sec - exiting 03:46:53 (1183): No heartbeat from core client for 30 sec - exiting 03:46:54 (1183): No heartbeat from core client for 30 sec - exiting 03:46:55 (1183): No heartbeat from core client for 30 sec - exiting 03:46:56 (1183): No heartbeat from core client for 30 sec - exiting 03:46:57 (1183): No heartbeat from core client for 30 sec - exiting 03:46:58 (1183): No heartbeat from core client for 30 sec - exiting 03:46:59 (1183): No heartbeat from core client for 30 sec - exiting 03:47:02 (1183): No heartbeat from core client for 30 sec - exiting 03:47:03 (1183): No heartbeat from core client for 30 sec - exiting 03:47:04 (1183): No heartbeat from core client for 30 sec - exiting 03:47:05 (1183): No heartbeat from core client for 30 sec - exiting 03:47:06 (1183): No heartbeat from core client for 30 sec - exiting 03:47:07 (1183): No heartbeat from core client for 30 sec - exiting 03:47:08 (1183): No heartbeat from core client for 30 sec - exiting 03:47:09 (1183): No heartbeat from core client for 30 sec - exiting 03:47:10 (1183): No heartbeat from core client for 30 sec - exiting 03:47:11 (1183): No heartbeat from core client for 30 sec - exiting 03:47:12 (1183): No heartbeat from core client for 30 sec - exiting 03:47:13 (1183): No heartbeat from core client for 30 sec - exiting 03:47:14 (1183): No heartbeat from core client for 30 sec - exiting 03:47:15 (1183): No heartbeat from core client for 30 sec - exiting 03:47:16 (1183): No heartbeat from core client for 30 sec - exiting 03:47:17 (1183): No heartbeat from core client for 30 sec - exiting 03:47:18 (1183): No heartbeat from core client for 30 sec - exiting 03:47:19 (1183): No heartbeat from core client for 30 sec - exiting 03:47:20 (1183): No heartbeat from core client for 30 sec - exiting 03:47:21 (1183): No heartbeat from core client for 30 sec - exiting 03:47:22 (1183): No heartbeat from core client for 30 sec - exiting 03:47:23 (1183): No heartbeat from core client for 30 sec - exiting 03:47:43 (1183): No heartbeat from core client for 30 sec - exiting 03:47:44 (1183): No heartbeat from core client for 30 sec - exiting 03:47:45 (1183): No heartbeat from core client for 30 sec - exiting 03:47:46 (1183): No heartbeat from core client for 30 sec - exiting 03:47:47 (1183): No heartbeat from core client for 30 sec - exiting 03:47:48 (1183): No heartbeat from core client for 30 sec - exiting 03:47:49 (1183): No heartbeat from core client for 30 sec - exiting 03:47:50 (1183): No heartbeat from core client for 30 sec - exiting 03:47:51 (1183): No heartbeat from core client for 30 sec - exiting 03:47:52 (1183): No heartbeat from core client for 30 sec - exiting 03:47:53 (1183): No heartbeat from core client for 30 sec - exiting 03:47:54 (1183): No heartbeat from core client for 30 sec - exiting 03:47:55 (1183): No heartbeat from core client for 30 sec - exiting 03:47:56 (1183): No heartbeat from core client for 30 sec - exiting 03:47:57 (1183): No heartbeat from core client for 30 sec - exiting 03:47:58 (1183): No heartbeat from core client for 30 sec - exiting 03:47:59 (1183): No heartbeat from core client for 30 sec - exiting 07:17:03 (1183): No heartbeat from core client for 30 sec - exiting 07:42:27 (1183): No heartbeat from core client for 30 sec - exiting 08:33:36 (1183): No heartbeat from core client for 30 sec - exiting 08:33:37 (1183): No heartbeat from core client for 30 sec - exiting 08:33:38 (1183): No heartbeat from core client for 30 sec - exiting 08:33:39 (1183): No heartbeat from core client for 30 sec - exiting 08:33:40 (1183): No heartbeat from core client for 30 sec - exiting 08:33:41 (1183): No heartbeat from core client for 30 sec - exiting 08:33:42 (1183): No heartbeat from core client for 30 sec - exiting 08:33:43 (1183): No heartbeat from core client for 30 sec - exiting 08:33:44 (1183): No heartbeat from core client for 30 sec - exiting 08:33:45 (1183): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09ff6318 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called execv: No such file or directory </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Sep 2012 15:01:11	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	492,480	2,478,896	5.0335
24 Aug 2012 10:26:35	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	259,200	1,291,170	4.9814
22 Aug 2012 07:31:04	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	233,280	1,152,457	4.9402
20 Aug 2012 10:44:51	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	207,360	1,016,525	4.9022
18 Aug 2012 20:51:35	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	181,440	882,506	4.8639
17 Aug 2012 01:53:35	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	155,520	752,044	4.8357
15 Aug 2012 05:04:17	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	129,600	622,146	4.8005
13 Aug 2012 16:11:49	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	103,680	494,378	4.7683
11 Aug 2012 21:02:35	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	77,760	365,880	4.7052
10 Aug 2012 05:01:09	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	51,840	235,774	4.5481
08 Aug 2012 07:56:24	1179185	15071465	hadcm3n_006o_1980_40_008000673_2	25,920	107,279	4.1389