Name | hadcm3n_o52a_2060_40_008409980_4 |
Workunit | 8560836 |
Created | 20 Dec 2013, 6:16:51 UTC |
Sent | 20 Dec 2013, 6:17:04 UTC |
Report deadline | 21 Mar 2014, 13:44:15 UTC |
Received | 15 May 2014, 4:31:10 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Aborted by user |
Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
Computer ID | 1292525 |
Run time | 69 days 23 hours 47 min 29 sec |
CPU time | 12 days 8 hours 7 min 16 sec |
Validate state | Invalid |
Credit | 3,421.44 |
Device peak FLOPS | 1.02 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
Stderr | <core_client_version>7.2.7</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> elfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... *** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0a0af680 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 04:24:52 (670): No heartbeat from core client for 30 sec - exiting 04:24:53 (670): No heartbeat from core client for 30 sec - exiting 04:24:54 (670): No heartbeat from core client for 30 sec - exiting 04:24:55 (670): No heartbeat from core client for 30 sec - exiting 04:24:56 (670): No heartbeat from core client for 30 sec - exiting 04:24:57 (670): No heartbeat from core client for 30 sec - exiting 04:24:58 (670): No heartbeat from core client for 30 sec - exiting 04:24:59 (670): No heartbeat from core client for 30 sec - exiting 04:25:00 (670): No heartbeat from core client for 30 sec - exiting 04:25:01 (670): No heartbeat from core client for 30 sec - exiting 04:25:02 (670): No heartbeat from core client for 30 sec - exiting 04:25:03 (670): No heartbeat from core client for 30 sec - exiting 04:25:04 (670): No heartbeat from core client for 30 sec - exiting 04:25:05 (670): No heartbeat from core client for 30 sec - exiting 04:25:06 (670): No heartbeat from core client for 30 sec - exiting 04:25:07 (670): No heartbeat from core client for 30 sec - exiting 04:25:08 (670): No heartbeat from core client for 30 sec - exiting 04:25:09 (670): No heartbeat from core client for 30 sec - exiting 04:25:10 (670): No heartbeat from core client for 30 sec - exiting 04:25:11 (670): No heartbeat from core client for 30 sec - exiting 04:25:12 (670): No heartbeat from core client for 30 sec - exiting 04:25:13 (670): No heartbeat from core client for 30 sec - exiting 04:25:14 (670): No heartbeat from core client for 30 sec - exiting 04:25:15 (670): No heartbeat from core client for 30 sec - exiting 04:25:16 (670): No heartbeat from core client for 30 sec - exiting 04:25:17 (670): No heartbeat from core client for 30 sec - exiting 04:25:18 (670): No heartbeat from core client for 30 sec - exiting 04:25:19 (670): No heartbeat from core client for 30 sec - exiting 04:25:20 (670): No heartbeat from core client for 30 sec - exiting 04:25:21 (670): No heartbeat from core client for 30 sec - exiting 04:25:22 (670): No heartbeat from core client for 30 sec - exiting 04:25:23 (670): No heartbeat from core client for 30 sec - exiting 04:25:24 (670): No heartbeat from core client for 30 sec - exiting 04:25:25 (670): No heartbeat from core client for 30 sec - exiting 04:25:26 (670): No heartbeat from core client for 30 sec - exiting 04:25:27 (670): No heartbeat from core client for 30 sec - exiting 04:25:28 (670): No heartbeat from core client for 30 sec - exiting 04:25:29 (670): No heartbeat from core client for 30 sec - exiting 04:25:30 (670): No heartbeat from core client for 30 sec - exiting 04:25:31 (670): No heartbeat from core client for 30 sec - exiting 04:25:32 (670): No heartbeat from core client for 30 sec - exiting 04:25:33 (670): No heartbeat from core client for 30 sec - exiting 04:25:34 (670): No heartbeat from core client for 30 sec - exiting 04:25:35 (670): No heartbeat from core client for 30 sec - exiting 04:25:36 (670): No heartbeat from core client for 30 sec - exiting 04:25:37 (670): No heartbeat from core client for 30 sec - exiting 04:25:38 (670): No heartbeat from core client for 30 sec - exiting 04:25:39 (670): No heartbeat from core client for 30 sec - exiting 04:25:40 (670): No heartbeat from core client for 30 sec - exiting 04:25:41 (670): No heartbeat from core client for 30 sec - exiting 04:25:42 (670): No heartbeat from core client for 30 sec - exiting 04:25:43 (670): No heartbeat from core client for 30 sec - exiting 04:25:44 (670): No heartbeat from core client for 30 sec - exiting 04:25:45 (670): No heartbeat from core client for 30 sec - exiting 04:25:46 (670): No heartbeat from core client for 30 sec - exiting 04:25:47 (670): No heartbeat from core client for 30 sec - exiting 04:25:48 (670): No heartbeat from core client for 30 sec - exiting 04:25:49 (670): No heartbeat from core client for 30 sec - exiting 04:25:50 (670): No heartbeat from core client for 30 sec - exiting 04:25:51 (670): No heartbeat from core client for 30 sec - exiting 04:25:52 (670): No heartbeat from core client for 30 sec - exiting 04:25:53 (670): No heartbeat from core client for 30 sec - exiting 04:25:54 (670): No heartbeat from core client for 30 sec - exiting 04:25:55 (670): No heartbeat from core client for 30 sec - exiting 04:25:56 (670): No heartbeat from core client for 30 sec - exiting 04:25:57 (670): No heartbeat from core client for 30 sec - exiting 04:25:58 (670): No heartbeat from core client for 30 sec - exiting 04:25:59 (670): No heartbeat from core client for 30 sec - exiting 04:26:00 (670): No heartbeat from core client for 30 sec - exiting 04:26:01 (670): No heartbeat from core client for 30 sec - exiting 04:26:02 (670): No heartbeat from core client for 30 sec - exiting 04:26:03 (670): No heartbeat from core client for 30 sec - exiting 04:26:04 (670): No heartbeat from core client for 30 sec - exiting 04:26:05 (670): No heartbeat from core client for 30 sec - exiting 04:26:06 (670): No heartbeat from core client for 30 sec - exiting 04:26:07 (670): No heartbeat from core client for 30 sec - exiting 04:26:08 (670): No heartbeat from core client for 30 sec - exiting 04:26:09 (670): No heartbeat from core client for 30 sec - exiting 04:26:10 (670): No heartbeat from core client for 30 sec - exiting 04:26:11 (670): No heartbeat from core client for 30 sec - exiting 04:26:12 (670): No heartbeat from core client for 30 sec - exiting 04:26:13 (670): No heartbeat from core client for 30 sec - exiting 04:26:14 (670): No heartbeat from core client for 30 sec - exiting 04:26:15 (670): No heartbeat from core client for 30 sec - exiting 04:26:16 (670): No heartbeat from core client for 30 sec - exiting 04:26:17 (670): No heartbeat from core client for 30 sec - exiting 04:26:18 (670): No heartbeat from core client for 30 sec - exiting 04:26:19 (670): No heartbeat from core client for 30 sec - exiting 04:26:20 (670): No heartbeat from core client for 30 sec - exiting 04:26:21 (670): No heartbeat from core client for 30 sec - exiting 04:26:22 (670): No heartbeat from core client for 30 sec - exiting 04:26:23 (670): No heartbeat from core client for 30 sec - exiting 04:26:24 (670): No heartbeat from core client for 30 sec - exiting 04:26:25 (670): No heartbeat from core client for 30 sec - exiting 04:26:26 (670): No heartbeat from core client for 30 sec - exiting 04:26:27 (670): No heartbeat from core client for 30 sec - exiting 04:26:28 (670): No heartbeat from core client for 30 sec - exiting 04:26:29 (670): No heartbeat from core client for 30 sec - exiting 04:26:30 (670): No heartbeat from core client for 30 sec - exiting 04:26:31 (670): No heartbeat from core client for 30 sec - exiting 04:26:32 (670): No heartbeat from core client for 30 sec - exiting 04:26:33 (670): No heartbeat from core client for 30 sec - exiting 04:26:34 (670): No heartbeat from core client for 30 sec - exiting 04:26:35 (670): No heartbeat from core client for 30 sec - exiting 04:26:36 (670): No heartbeat from core client for 30 sec - exiting 04:26:37 (670): No heartbeat from core client for 30 sec - exiting 04:26:38 (670): No heartbeat from core client for 30 sec - exiting 04:26:39 (670): No heartbeat from core client for 30 sec - exiting 04:26:40 (670): No heartbeat from core client for 30 sec - exiting 04:26:41 (670): No heartbeat from core client for 30 sec - exiting 04:26:42 (670): No heartbeat from core client for 30 sec - exiting 04:26:43 (670): No heartbeat from core client for 30 sec - exiting 04:26:44 (670): No heartbeat from core client for 30 sec - exiting 04:26:45 (670): No heartbeat from core client for 30 sec - exiting 04:26:46 (670): No heartbeat from core client for 30 sec - exiting 04:26:47 (670): No heartbeat from core client for 30 sec - exiting 04:26:48 (670): No heartbeat from core client for 30 sec - exiting 04:26:49 (670): No heartbeat from core client for 30 sec - exiting 04:26:50 (670): No heartbeat from core client for 30 sec - exiting 04:26:51 (670): No heartbeat from core client for 30 sec - exiting 04:26:52 (670): No heartbeat from core client for 30 sec - exiting 04:26:53 (670): No heartbeat from core client for 30 sec - exiting 04:26:54 (670): No heartbeat from core client for 30 sec - exiting 04:26:55 (670): No heartbeat from core client for 30 sec - exiting 04:26:56 (670): No heartbeat from core client for 30 sec - exiting 04:26:57 (670): No heartbeat from core client for 30 sec - exiting 04:26:58 (670): No heartbeat from core client for 30 sec - exiting 04:26:59 (670): No heartbeat from core client for 30 sec - exiting 04:27:00 (670): No heartbeat from core client for 30 sec - exiting 04:27:01 (670): No heartbeat from core client for 30 sec - exiting 04:27:02 (670): No heartbeat from core client for 30 sec - exiting 04:27:03 (670): No heartbeat from core client for 30 sec - exiting 04:27:04 (670): No heartbeat from core client for 30 sec - exiting 04:27:05 (670): No heartbeat from core client for 30 sec - exiting 04:27:06 (670): No heartbeat from core client for 30 sec - exiting 04:27:07 (670): No heartbeat from core client for 30 sec - exiting 04:27:08 (670): No heartbeat from core client for 30 sec - exiting 04:27:09 (670): No heartbeat from core client for 30 sec - exiting 04:27:10 (670): No heartbeat from core client for 30 sec - exiting 04:27:11 (670): No heartbeat from core client for 30 sec - exiting 04:27:12 (670): No heartbeat from core client for 30 sec - exiting 04:27:13 (670): No heartbeat from core client for 30 sec - exiting 04:27:14 (670): No heartbeat from core client for 30 sec - exiting 04:27:16 (670): No heartbeat from core client for 30 sec - exiting 04:27:17 (670): No heartbeat from core client for 30 sec - exiting 04:27:18 (670): No heartbeat from core client for 30 sec - exiting 04:27:19 (670): No heartbeat from core client for 30 sec - exiting 04:27:20 (670): No heartbeat from core client for 30 sec - exiting 04:27:21 (670): No heartbeat from core client for 30 sec - exiting 04:27:22 (670): No heartbeat from core client for 30 sec - exiting 04:27:23 (670): No heartbeat from core client for 30 sec - exiting 04:27:24 (670): No heartbeat from core client for 30 sec - exiting 04:27:25 (670): No heartbeat from core client for 30 sec - exiting 04:27:26 (670): No heartbeat from core client for 30 sec - exiting 04:27:27 (670): No heartbeat from core client for 30 sec - exiting 04:27:28 (670): No heartbeat from core client for 30 sec - exiting 04:27:29 (670): No heartbeat from core client for 30 sec - exiting 04:27:30 (670): No heartbeat from core client for 30 sec - exiting 04:27:31 (670): No heartbeat from core client for 30 sec - exiting 04:33:26 (670): No heartbeat from core client for 30 sec - exiting 04:33:27 (670): No heartbeat from core client for 30 sec - exiting 04:33:28 (670): No heartbeat from core client for 30 sec - exiting 04:33:29 (670): No heartbeat from core client for 30 sec - exiting 04:33:30 (670): No heartbeat from core client for 30 sec - exiting 04:33:31 (670): No heartbeat from core client for 30 sec - exiting 04:33:32 (670): No heartbeat from core client for 30 sec - exiting 04:33:33 (670): No heartbeat from core client for 30 sec - exiting 04:33:34 (670): No heartbeat from core client for 30 sec - exiting 04:33:35 (670): No heartbeat from core client for 30 sec - exiting 04:33:36 (670): No heartbeat from core client for 30 sec - exiting 04:33:37 (670): No heartbeat from core client for 30 sec - exiting 04:33:38 (670): No heartbeat from core client for 30 sec - exiting 04:33:39 (670): No heartbeat from core client for 30 sec - exiting 04:33:40 (670): No heartbeat from core client for 30 sec - exiting 04:33:41 (670): No heartbeat from core client for 30 sec - exiting 04:33:42 (670): No heartbeat from core client for 30 sec - exiting 04:33:43 (670): No heartbeat from core client for 30 sec - exiting 04:33:44 (670): No heartbeat from core client for 30 sec - exiting 04:33:45 (670): No heartbeat from core client for 30 sec - exiting 04:33:46 (670): No heartbeat from core client for 30 sec - exiting 04:33:47 (670): No heartbeat from core client for 30 sec - exiting 04:33:48 (670): No heartbeat from core client for 30 sec - exiting 04:33:49 (670): No heartbeat from core client for 30 sec - exiting 04:33:50 (670): No heartbeat from core client for 30 sec - exiting 04:33:51 (670): No heartbeat from core client for 30 sec - exiting 04:33:52 (670): No heartbeat from core client for 30 sec - exiting 04:33:53 (670): No heartbeat from core client for 30 sec - exiting 04:33:54 (670): No heartbeat from core client for 30 sec - exiting 04:33:55 (670): No heartbeat from core client for 30 sec - exiting 04:33:56 (670): No heartbeat from core client for 30 sec - exiting 04:33:57 (670): No heartbeat from core client for 30 sec - exiting 04:33:58 (670): No heartbeat from core client for 30 sec - exiting 04:33:59 (670): No heartbeat from core client for 30 sec - exiting 04:34:00 (670): No heartbeat from core client for 30 sec - exiting 04:34:01 (670): No heartbeat from core client for 30 sec - exiting 04:34:02 (670): No heartbeat from core client for 30 sec - exiting 04:34:03 (670): No heartbeat from core client for 30 sec - exiting 04:34:04 (670): No heartbeat from core client for 30 sec - exiting 04:34:05 (670): No heartbeat from core client for 30 sec - exiting 04:34:06 (670): No heartbeat from core client for 30 sec - exiting 04:34:07 (670): No heartbeat from core client for 30 sec - exiting 04:34:08 (670): No heartbeat from core client for 30 sec - exiting 04:34:09 (670): No heartbeat from core client for 30 sec - exiting 04:34:10 (670): No heartbeat from core client for 30 sec - exiting 04:34:11 (670): No heartbeat from core client for 30 sec - exiting 04:34:12 (670): No heartbeat from core client for 30 sec - exiting 04:34:13 (670): No heartbeat from core client for 30 sec - exiting 04:34:14 (670): No heartbeat from core client for 30 sec - exiting 04:34:15 (670): No heartbeat from core client for 30 sec - exiting 04:34:16 (670): No heartbeat from core client for 30 sec - exiting 04:34:17 (670): No heartbeat from core client for 30 sec - exiting 04:34:18 (670): No heartbeat from core client for 30 sec - exiting 04:34:19 (670): No heartbeat from core client for 30 sec - exiting 04:34:20 (670): No heartbeat from core client for 30 sec - exiting 04:34:21 (670): No heartbeat from core client for 30 sec - exiting 04:34:22 (670): No heartbeat from core client for 30 sec - exiting 04:34:23 (670): No heartbeat from core client for 30 sec - exiting 04:34:24 (670): No heartbeat from core client for 30 sec - exiting 04:34:25 (670): No heartbeat from core client for 30 sec - exiting 04:34:26 (670): No heartbeat from core client for 30 sec - exiting 04:34:27 (670): No heartbeat from core client for 30 sec - exiting 04:34:28 (670): No heartbeat from core client for 30 sec - exiting 04:34:29 (670): No heartbeat from core client for 30 sec - exiting 04:34:30 (670): No heartbeat from core client for 30 sec - exiting 04:34:31 (670): No heartbeat from core client for 30 sec - exiting 04:34:32 (670): No heartbeat from core client for 30 sec - exiting 04:34:33 (670): No heartbeat from core client for 30 sec - exiting 04:34:34 (670): No heartbeat from core client for 30 sec - exiting 04:34:35 (670): No heartbeat from core client for 30 sec - exiting 04:34:36 (670): No heartbeat from core client for 30 sec - exiting 04:34:37 (670): No heartbeat from core client for 30 sec - exiting 04:34:38 (670): No heartbeat from core client for 30 sec - exiting 04:34:39 (670): No heartbeat from core client for 30 sec - exiting 04:34:40 (670): No heartbeat from core client for 30 sec - exiting 04:34:41 (670): No heartbeat from core client for 30 sec - exiting 04:34:42 (670): No heartbeat from core client for 30 sec - exiting 04:34:43 (670): No heartbeat from core client for 30 sec - exiting 04:34:44 (670): No heartbeat from core client for 30 sec - exiting 04:34:45 (670): No heartbeat from core client for 30 sec - exiting 04:34:46 (670): No heartbeat from core client for 30 sec - exiting 04:34:47 (670): No heartbeat from core client for 30 sec - exiting 04:34:48 (670): No heartbeat from core client for 30 sec - exiting 04:34:49 (670): No heartbeat from core client for 30 sec - exiting 04:34:50 (670): No heartbeat from core client for 30 sec - exiting 04:34:51 (670): No heartbeat from core client for 30 sec - exiting 04:34:52 (670): No heartbeat from core client for 30 sec - exiting 04:34:53 (670): No heartbeat from core client for 30 sec - exiting 04:34:54 (670): No heartbeat from core client for 30 sec - exiting 04:34:55 (670): No heartbeat from core client for 30 sec - exiting 04:34:56 (670): No heartbeat from core client for 30 sec - exiting 04:34:57 (670): No heartbeat from core client for 30 sec - exiting 04:34:58 (670): No heartbeat from core client for 30 sec - exiting 04:34:59 (670): No heartbeat from core client for 30 sec - exiting 04:35:00 (670): No heartbeat from core client for 30 sec - exiting 04:35:01 (670): No heartbeat from core client for 30 sec - exiting 04:35:02 (670): No heartbeat from core client for 30 sec - exiting 04:35:03 (670): No heartbeat from core client for 30 sec - exiting 04:35:04 (670): No heartbeat from core client for 30 sec - exiting 04:35:05 (670): No heartbeat from core client for 30 sec - exiting 04:35:06 (670): No heartbeat from core client for 30 sec - exiting 04:35:07 (670): No heartbeat from core client for 30 sec - exiting 04:35:08 (670): No heartbeat from core client for 30 sec - exiting 04:35:09 (670): No heartbeat from core client for 30 sec - exiting 04:35:10 (670): No heartbeat from core client for 30 sec - exiting 04:35:11 (670): No heartbeat from core client for 30 sec - exiting 04:35:12 (670): No heartbeat from core client for 30 sec - exiting 04:35:13 (670): No heartbeat from core client for 30 sec - exiting 04:35:14 (670): No heartbeat from core client for 30 sec - exiting 04:35:15 (670): No heartbeat from core client for 30 sec - exiting 04:35:16 (670): No heartbeat from core client for 30 sec - exiting 04:35:17 (670): No heartbeat from core client for 30 sec - exiting 04:35:18 (670): No heartbeat from core client for 30 sec - exiting 04:35:19 (670): No heartbeat from core client for 30 sec - exiting 04:35:20 (670): No heartbeat from core client for 30 sec - exiting 04:35:21 (670): No heartbeat from core client for 30 sec - exiting 04:35:22 (670): No heartbeat from core client for 30 sec - exiting 04:35:23 (670): No heartbeat from core client for 30 sec - exiting 04:35:24 (670): No heartbeat from core client for 30 sec - exiting 04:35:25 (670): No heartbeat from core client for 30 sec - exiting 04:35:26 (670): No heartbeat from core client for 30 sec - exiting 04:35:27 (670): No heartbeat from core client for 30 sec - exiting 04:35:28 (670): No heartbeat from core client for 30 sec - exiting 04:35:29 (670): No heartbeat from core client for 30 sec - exiting 04:35:30 (670): No heartbeat from core client for 30 sec - exiting 04:35:31 (670): No heartbeat from core client for 30 sec - exiting 04:35:32 (670): No heartbeat from core client for 30 sec - exiting 04:35:33 (670): No heartbeat from core client for 30 sec - exiting 04:35:34 (670): No heartbeat from core client for 30 sec - exiting 04:35:35 (670): No heartbeat from core client for 30 sec - exiting 04:35:36 (670): No heartbeat from core client for 30 sec - exiting 04:35:37 (670): No heartbeat from core client for 30 sec - exiting 04:35:38 (670): No heartbeat from core client for 30 sec - exiting 04:35:39 (670): No heartbeat from core client for 30 sec - exiting 04:35:40 (670): No heartbeat from core client for 30 sec - exiting 04:35:41 (670): No heartbeat from core client for 30 sec - exiting 04:35:42 (670): No heartbeat from core client for 30 sec - exiting 04:35:43 (670): No heartbeat from core client for 30 sec - exiting 04:35:44 (670): No heartbeat from core client for 30 sec - exiting 04:35:45 (670): No heartbeat from core client for 30 sec - exiting 04:35:46 (670): No heartbeat from core client for 30 sec - exiting 04:35:47 (670): No heartbeat from core client for 30 sec - exiting 04:35:48 (670): No heartbeat from core client for 30 sec - exiting 04:35:49 (670): No heartbeat from core client for 30 sec - exiting 04:35:50 (670): No heartbeat from core client for 30 sec - exiting 04:35:51 (670): No heartbeat from core client for 30 sec - exiting 04:35:52 (670): No heartbeat from core client for 30 sec - exiting 04:35:53 (670): No heartbeat from core client for 30 sec - exiting 04:35:54 (670): No heartbeat from core client for 30 sec - exiting 04:35:55 (670): No heartbeat from core client for 30 sec - exiting 04:35:56 (670): No heartbeat from core client for 30 sec - exiting 04:35:57 (670): No heartbeat from core client for 30 sec - exiting 04:35:58 (670): No heartbeat from core client for 30 sec - exiting 04:35:59 (670): No heartbeat from core client for 30 sec - exiting 04:36:00 (670): No heartbeat from core client for 30 sec - exiting 04:36:01 (670): No heartbeat from core client for 30 sec - exiting 04:36:02 (670): No heartbeat from core client for 30 sec - exiting 04:36:03 (670): No heartbeat from core client for 30 sec - exiting 04:43:53 (670): No heartbeat from core client for 30 sec - exiting 04:43:54 (670): No heartbeat from core client for 30 sec - exiting 04:43:55 (670): No heartbeat from core client for 30 sec - exiting 04:43:56 (670): No heartbeat from core client for 30 sec - exiting 04:43:57 (670): No heartbeat from core client for 30 sec - exiting 04:43:58 (670): No heartbeat from core client for 30 sec - exiting 04:43:59 (670): No heartbeat from core client for 30 sec - exiting 04:44:00 (670): No heartbeat from core client for 30 sec - exiting 04:44:01 (670): No heartbeat from core client for 30 sec - exiting 04:44:02 (670): No heartbeat from core client for 30 sec - exiting 04:44:03 (670): No heartbeat from core client for 30 sec - exiting 04:44:04 (670): No heartbeat from core client for 30 sec - exiting 04:44:05 (670): No heartbeat from core client for 30 sec - exiting 04:44:06 (670): No heartbeat from core client for 30 sec - exiting 04:44:07 (670): No heartbeat from core client for 30 sec - exiting 04:44:08 (670): No heartbeat from core client for 30 sec - exiting 04:44:09 (670): No heartbeat from core client for 30 sec - exiting 04:44:10 (670): No heartbeat from core client for 30 sec - exiting 04:44:11 (670): No heartbeat from core client for 30 sec - exiting 04:44:12 (670): No heartbeat from core client for 30 sec - exiting 04:44:13 (670): No heartbeat from core client for 30 sec - exiting 04:44:14 (670): No heartbeat from core client for 30 sec - exiting 04:44:15 (670): No heartbeat from core client for 30 sec - exiting 04:44:16 (670): No heartbeat from core client for 30 sec - exiting 04:44:17 (670): No heartbeat from core client for 30 sec - exiting 04:44:18 (670): No heartbeat from core client for 30 sec - exiting 04:44:19 (670): No heartbeat from core client for 30 sec - exiting 04:44:20 (670): No heartbeat from core client for 30 sec - exiting 04:44:21 (670): No heartbeat from core client for 30 sec - exiting 04:44:22 (670): No heartbeat from core client for 30 sec - exiting 04:44:23 (670): No heartbeat from core client for 30 sec - exiting 04:44:24 (670): No heartbeat from core client for 30 sec - exiting 04:44:25 (670): No heartbeat from core client for 30 sec - exiting 04:44:26 (670): No heartbeat from core client for 30 sec - exiting 04:44:27 (670): No heartbeat from core client for 30 sec - exiting 04:44:28 (670): No heartbeat from core client for 30 sec - exiting 04:44:29 (670): No heartbeat from core client for 30 sec - exiting 04:44:30 (670): No heartbeat from core client for 30 sec - exiting 04:44:31 (670): No heartbeat from core client for 30 sec - exiting 04:44:32 (670): No heartbeat from core client for 30 sec - exiting 04:44:33 (670): No heartbeat from core client for 30 sec - exiting 04:44:34 (670): No heartbeat from core client for 30 sec - exiting 04:44:35 (670): No heartbeat from core client for 30 sec - exiting 04:44:36 (670): No heartbeat from core client for 30 sec - exiting 04:44:37 (670): No heartbeat from core client for 30 sec - exiting 04:44:38 (670): No heartbeat from core client for 30 sec - exiting 04:44:39 (670): No heartbeat from core client for 30 sec - exiting 04:44:40 (670): No heartbeat from core client for 30 sec - exiting 04:44:41 (670): No heartbeat from core client for 30 sec - exiting 04:44:42 (670): No heartbeat from core client for 30 sec - exiting 04:44:43 (670): No heartbeat from core client for 30 sec - exiting 04:44:44 (670): No heartbeat from core client for 30 sec - exiting 04:44:45 (670): No heartbeat from core client for 30 sec - exiting 04:44:46 (670): No heartbeat from core client for 30 sec - exiting 04:44:47 (670): No heartbeat from core client for 30 sec - exiting 04:44:48 (670): No heartbeat from core client for 30 sec - exiting 04:44:49 (670): No heartbeat from core client for 30 sec - exiting 04:44:50 (670): No heartbeat from core client for 30 sec - exiting 04:44:51 (670): No heartbeat from core client for 30 sec - exiting 04:44:52 (670): No heartbeat from core client for 30 sec - exiting 04:44:53 (670): No heartbeat from core client for 30 sec - exiting 04:44:54 (670): No heartbeat from core client for 30 sec - exiting 04:44:55 (670): No heartbeat from core client for 30 sec - exiting 04:44:56 (670): No heartbeat from core client for 30 sec - exiting 04:44:57 (670): No heartbeat from core client for 30 sec - exiting 04:44:58 (670): No heartbeat from core client for 30 sec - exiting 04:44:59 (670): No heartbeat from core client for 30 sec - exiting 04:45:00 (670): No heartbeat from core client for 30 sec - exiting 04:45:01 (670): No heartbeat from core client for 30 sec - exiting 04:45:02 (670): No heartbeat from core client for 30 sec - exiting 04:45:03 (670): No heartbeat from core client for 30 sec - exiting 04:45:04 (670): No heartbeat from core client for 30 sec - exiting 04:45:05 (670): No heartbeat from core client for 30 sec - exiting 04:45:06 (670): No heartbeat from core client for 30 sec - exiting 04:45:07 (670): No heartbeat from core client for 30 sec - exiting 04:45:08 (670): No heartbeat from core client for 30 sec - exiting 04:45:09 (670): No heartbeat from core client for 30 sec - exiting 04:45:10 (670): No heartbeat from core client for 30 sec - exiting 04:45:11 (670): No heartbeat from core client for 30 sec - exiting 04:45:12 (670): No heartbeat from core client for 30 sec - exiting 04:45:13 (670): No heartbeat from core client for 30 sec - exiting 04:45:14 (670): No heartbeat from core client for 30 sec - exiting 04:45:15 (670): No heartbeat from core client for 30 sec - exiting 04:45:16 (670): No heartbeat from core client for 30 sec - exiting 04:45:17 (670): No heartbeat from core client for 30 sec - exiting 04:45:18 (670): No heartbeat from core client for 30 sec - exiting 04:45:19 (670): No heartbeat from core client for 30 sec - exiting 04:45:20 (670): No heartbeat from core client for 30 sec - exiting 04:45:21 (670): No heartbeat from core client for 30 sec - exiting 04:45:22 (670): No heartbeat from core client for 30 sec - exiting 04:45:23 (670): No heartbeat from core client for 30 sec - exiting 04:45:24 (670): No heartbeat from core client for 30 sec - exiting 04:45:25 (670): No heartbeat from core client for 30 sec - exiting 04:45:26 (670): No heartbeat from core client for 30 sec - exiting 04:45:27 (670): No heartbeat from core client for 30 sec - exiting 04:45:28 (670): No heartbeat from core client for 30 sec - exiting 04:45:29 (670): No heartbeat from core client for 30 sec - exiting 04:45:30 (670): No heartbeat from core client for 30 sec - exiting 04:45:31 (670): No heartbeat from core client for 30 sec - exiting 04:45:32 (670): No heartbeat from core client for 30 sec - exiting 04:45:33 (670): No heartbeat from core client for 30 sec - exiting 04:45:34 (670): No heartbeat from core client for 30 sec - exiting 04:45:35 (670): No heartbeat from core client for 30 sec - exiting 04:45:36 (670): No heartbeat from core client for 30 sec - exiting 04:45:37 (670): No heartbeat from core client for 30 sec - exiting 04:45:38 (670): No heartbeat from core client for 30 sec - exiting 04:45:39 (670): No heartbeat from core client for 30 sec - exiting 04:45:40 (670): No heartbeat from core client for 30 sec - exiting 04:45:41 (670): No heartbeat from core client for 30 sec - exiting 04:45:42 (670): No heartbeat from core client for 30 sec - exiting 04:45:43 (670): No heartbeat from core client for 30 sec - exiting 04:45:44 (670): No heartbeat from core client for 30 sec - exiting 04:45:45 (670): No heartbeat from core client for 30 sec - exiting 04:45:46 (670): No heartbeat from core client for 30 sec - exiting 04:45:47 (670): No heartbeat from core client for 30 sec - exiting 04:45:48 (670): No heartbeat from core client for 30 sec - exiting 04:45:49 (670): No heartbeat from core client for 30 sec - exiting 04:45:50 (670): No heartbeat from core client for 30 sec - exiting 04:45:51 (670): No heartbeat from core client for 30 sec - exiting 04:45:52 (670): No heartbeat from core client for 30 sec - exiting 04:45:53 (670): No heartbeat from core client for 30 sec - exiting 04:45:54 (670): No heartbeat from core client for 30 sec - exiting 04:45:55 (670): No heartbeat from core client for 30 sec - exiting 04:45:56 (670): No heartbeat from core client for 30 sec - exiting 04:45:57 (670): No heartbeat from core client for 30 sec - exiting 04:45:58 (670): No heartbeat from core client for 30 sec - exiting 04:45:59 (670): No heartbeat from core client for 30 sec - exiting 04:46:00 (670): No heartbeat from core client for 30 sec - exiting 04:46:01 (670): No heartbeat from core client for 30 sec - exiting 04:46:02 (670): No heartbeat from core client for 30 sec - exiting 04:46:03 (670): No heartbeat from core client for 30 sec - exiting 04:46:04 (670): No heartbeat from core client for 30 sec - exiting 04:46:05 (670): No heartbeat from core client for 30 sec - exiting 04:46:06 (670): No heartbeat from core client for 30 sec - exiting 04:46:07 (670): No heartbeat from core client for 30 sec - exiting 04:46:08 (670): No heartbeat from core client for 30 sec - exiting 04:46:09 (670): No heartbeat from core client for 30 sec - exiting 04:46:10 (670): No heartbeat from core client for 30 sec - exiting 04:46:11 (670): No heartbeat from core client for 30 sec - exiting 04:46:12 (670): No heartbeat from core client for 30 sec - exiting 04:46:13 (670): No heartbeat from core client for 30 sec - exiting 04:46:14 (670): No heartbeat from core client for 30 sec - exiting 04:46:15 (670): No heartbeat from core client for 30 sec - exiting 04:46:16 (670): No heartbeat from core client for 30 sec - exiting 04:46:17 (670): No heartbeat from core client for 30 sec - exiting 04:46:18 (670): No heartbeat from core client for 30 sec - exiting 04:46:19 (670): No heartbeat from core client for 30 sec - exiting 04:46:20 (670): No heartbeat from core client for 30 sec - exiting 04:46:21 (670): No heartbeat from core client for 30 sec - exiting 04:46:22 (670): No heartbeat from core client for 30 sec - exiting 04:46:23 (670): No heartbeat from core client for 30 sec - exiting 04:46:24 (670): No heartbeat from core client for 30 sec - exiting 04:46:25 (670): No heartbeat from core client for 30 sec - exiting 04:46:26 (670): No heartbeat from core client for 30 sec - exiting 04:46:27 (670): No heartbeat from core client for 30 sec - exiting 04:46:28 (670): No heartbeat from core client for 30 sec - exiting 04:46:29 (670): No heartbeat from core client for 30 sec - exiting 04:46:30 (670): No heartbeat from core client for 30 sec - exiting *** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09681f78 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08b47f78 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x082c6f78 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08cb1fa8 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called *** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09a90f78 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
01 Jan 2014 22:12:37 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 285,120 | 1,066,036 | 3.7389 |
31 Dec 2013 12:10:35 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 259,200 | 947,444 | 3.6553 |
30 Dec 2013 09:57:15 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 233,280 | 853,241 | 3.6576 |
29 Dec 2013 07:12:02 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 207,360 | 757,191 | 3.6516 |
28 Dec 2013 04:19:31 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 181,440 | 660,559 | 3.6406 |
27 Dec 2013 01:25:06 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 155,520 | 563,782 | 3.6251 |
25 Dec 2013 21:51:06 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 129,600 | 464,791 | 3.5864 |
24 Dec 2013 15:50:46 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 103,680 | 370,346 | 3.5720 |
23 Dec 2013 13:57:33 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 77,760 | 278,199 | 3.5777 |
22 Dec 2013 10:34:54 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 51,840 | 184,598 | 3.5609 |
21 Dec 2013 08:11:14 | 1292525 | 16150414 | hadcm3n_o52a_2060_40_008409980_4 | 25,920 | 91,458 | 3.5285 |
©2024 cpdn.org