Task 13664065

Name	hadcm3n_u53j_1980_40_007539683_2
Workunit	7736915
Created	26 Nov 2011, 21:44:18 UTC
Sent	26 Nov 2011, 21:44:58 UTC
Report deadline	26 Feb 2012, 5:12:09 UTC
Received	12 Dec 2011, 3:09:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	13 days 11 hours 2 min 49 sec
CPU time	13 days 4 hours 20 min 15 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	1.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 02:29:49 (4472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:37:58 (5040): No heartbeat from core client for 30 sec - exiting 02:37:59 (5040): No heartbeat from core client for 30 sec - exiting 02:38:00 (5040): No heartbeat from core client for 30 sec - exiting 02:38:01 (5040): No heartbeat from core client for 30 sec - exiting 02:38:02 (5040): No heartbeat from core client for 30 sec - exiting 02:38:03 (5040): No heartbeat from core client for 30 sec - exiting 02:38:05 (5040): No heartbeat from core client for 30 sec - exiting 02:38:06 (5040): No heartbeat from core client for 30 sec - exiting 02:38:07 (5040): No heartbeat from core client for 30 sec - exiting 02:38:08 (5040): No heartbeat from core client for 30 sec - exiting 02:38:09 (5040): No heartbeat from core client for 30 sec - exiting 02:38:10 (5040): No heartbeat from core client for 30 sec - exiting 02:38:11 (5040): No heartbeat from core client for 30 sec - exiting 02:38:12 (5040): No heartbeat from core client for 30 sec - exiting 02:38:13 (5040): No heartbeat from core client for 30 sec - exiting 02:38:14 (5040): No heartbeat from core client for 30 sec - exiting 02:38:15 (5040): No heartbeat from core client for 30 sec - exiting 02:38:17 (5040): No heartbeat from core client for 30 sec - exiting 02:38:18 (5040): No heartbeat from core client for 30 sec - exiting 02:38:19 (5040): No heartbeat from core client for 30 sec - exiting 02:38:20 (5040): No heartbeat from core client for 30 sec - exiting 02:38:21 (5040): No heartbeat from core client for 30 sec - exiting 02:38:22 (5040): No heartbeat from core client for 30 sec - exiting 02:38:23 (5040): No heartbeat from core client for 30 sec - exiting 02:38:24 (5040): No heartbeat from core client for 30 sec - exiting 02:38:25 (5040): No heartbeat from core client for 30 sec - exiting 02:38:26 (5040): No heartbeat from core client for 30 sec - exiting 02:38:27 (5040): No heartbeat from core client for 30 sec - exiting 02:38:29 (5040): No heartbeat from core client for 30 sec - exiting 02:38:30 (5040): No heartbeat from core client for 30 sec - exiting 02:38:31 (5040): No heartbeat from core client for 30 sec - exiting 02:38:32 (5040): No heartbeat from core client for 30 sec - exiting 02:38:33 (5040): No heartbeat from core client for 30 sec - exiting 02:38:34 (5040): No heartbeat from core client for 30 sec - exiting 02:38:35 (5040): No heartbeat from core client for 30 sec - exiting 02:38:36 (5040): No heartbeat from core client for 30 sec - exiting 02:38:37 (5040): No heartbeat from core client for 30 sec - exiting 02:38:38 (5040): No heartbeat from core client for 30 sec - exiting 02:38:39 (5040): No heartbeat from core client for 30 sec - exiting 02:38:41 (5040): No heartbeat from core client for 30 sec - exiting 02:38:42 (5040): No heartbeat from core client for 30 sec - exiting 02:38:43 (5040): No heartbeat from core client for 30 sec - exiting 02:38:44 (5040): No heartbeat from core client for 30 sec - exiting 02:38:45 (5040): No heartbeat from core client for 30 sec - exiting 02:38:46 (5040): No heartbeat from core client for 30 sec - exiting 02:38:47 (5040): No heartbeat from core client for 30 sec - exiting 02:38:48 (5040): No heartbeat from core client for 30 sec - exiting 02:38:49 (5040): No heartbeat from core client for 30 sec - exiting 02:38:50 (5040): No heartbeat from core client for 30 sec - exiting 02:39:24 (5040): No heartbeat from core client for 30 sec - exiting 02:39:25 (5040): No heartbeat from core client for 30 sec - exiting 02:39:26 (5040): No heartbeat from core client for 30 sec - exiting 02:39:28 (5040): No heartbeat from core client for 30 sec - exiting 02:39:29 (5040): No heartbeat from core client for 30 sec - exiting 02:39:30 (5040): No heartbeat from core client for 30 sec - exiting 02:39:31 (5040): No heartbeat from core client for 30 sec - exiting 02:39:32 (5040): No heartbeat from core client for 30 sec - exiting 02:39:33 (5040): No heartbeat from core client for 30 sec - exiting 02:39:34 (5040): No heartbeat from core client for 30 sec - exiting 02:39:35 (5040): No heartbeat from core client for 30 sec - exiting 02:39:36 (5040): No heartbeat from core client for 30 sec - exiting 02:39:37 (5040): No heartbeat from core client for 30 sec - exiting 02:39:41 (5040): No heartbeat from core client for 30 sec - exiting 02:39:43 (5040): No heartbeat from core client for 30 sec - exiting 02:39:44 (5040): No heartbeat from core client for 30 sec - exiting 02:39:45 (5040): No heartbeat from core client for 30 sec - exiting 02:39:46 (5040): No heartbeat from core client for 30 sec - exiting 02:39:47 (5040): No heartbeat from core client for 30 sec - exiting 02:39:48 (5040): No heartbeat from core client for 30 sec - exiting 02:39:49 (5040): No heartbeat from core client for 30 sec - exiting 02:39:50 (5040): No heartbeat from core client for 30 sec - exiting 02:39:51 (5040): No heartbeat from core client for 30 sec - exiting 02:39:52 (5040): No heartbeat from core client for 30 sec - exiting 02:39:53 (5040): No heartbeat from core client for 30 sec - exiting 02:39:55 (5040): No heartbeat from core client for 30 sec - exiting 02:39:56 (5040): No heartbeat from core client for 30 sec - exiting 02:39:57 (5040): No heartbeat from core client for 30 sec - exiting 02:39:58 (5040): No heartbeat from core client for 30 sec - exiting 02:39:59 (5040): No heartbeat from core client for 30 sec - exiting 02:40:00 (5040): No heartbeat from core client for 30 sec - exiting 02:40:01 (5040): No heartbeat from core client for 30 sec - exiting 02:40:02 (5040): No heartbeat from core client for 30 sec - exiting 02:40:03 (5040): No heartbeat from core client for 30 sec - exiting 02:40:04 (5040): No heartbeat from core client for 30 sec - exiting 02:40:05 (5040): No heartbeat from core client for 30 sec - exiting 02:40:07 (5040): No heartbeat from core client for 30 sec - exiting 02:40:08 (5040): No heartbeat from core client for 30 sec - exiting 02:40:09 (5040): No heartbeat from core client for 30 sec - exiting 02:40:10 (5040): No heartbeat from core client for 30 sec - exiting 02:40:11 (5040): No heartbeat from core client for 30 sec - exiting 02:40:12 (5040): No heartbeat from core client for 30 sec - exiting 02:40:13 (5040): No heartbeat from core client for 30 sec - exiting 02:40:14 (5040): No heartbeat from core client for 30 sec - exiting 02:40:15 (5040): No heartbeat from core client for 30 sec - exiting 02:40:16 (5040): No heartbeat from core client for 30 sec - exiting 02:40:17 (5040): No heartbeat from core client for 30 sec - exiting 02:40:19 (5040): No heartbeat from core client for 30 sec - exiting 02:40:20 (5040): No heartbeat from core client for 30 sec - exiting 02:40:21 (5040): No heartbeat from core client for 30 sec - exiting 02:40:22 (5040): No heartbeat from core client for 30 sec - exiting 02:40:23 (5040): No heartbeat from core client for 30 sec - exiting 02:40:24 (5040): No heartbeat from core client for 30 sec - exiting 02:40:25 (5040): No heartbeat from core client for 30 sec - exiting 02:40:26 (5040): No heartbeat from core client for 30 sec - exiting 02:40:27 (5040): No heartbeat from core client for 30 sec - exiting 02:40:28 (5040): No heartbeat from core client for 30 sec - exiting 02:40:29 (5040): No heartbeat from core client for 30 sec - exiting 02:40:31 (5040): No heartbeat from core client for 30 sec - exiting 02:40:32 (5040): No heartbeat from core client for 30 sec - exiting 02:40:33 (5040): No heartbeat from core client for 30 sec - exiting 02:40:34 (5040): No heartbeat from core client for 30 sec - exiting 02:40:35 (5040): No heartbeat from core client for 30 sec - exiting 02:41:09 (5040): No heartbeat from core client for 30 sec - exiting 02:41:10 (5040): No heartbeat from core client for 30 sec - exiting 02:41:11 (5040): No heartbeat from core client for 30 sec - exiting 02:41:12 (5040): No heartbeat from core client for 30 sec - exiting 02:41:13 (5040): No heartbeat from core client for 30 sec - exiting 02:41:14 (5040): No heartbeat from core client for 30 sec - exiting 02:41:15 (5040): No heartbeat from core client for 30 sec - exiting 02:41:16 (5040): No heartbeat from core client for 30 sec - exiting 02:41:18 (5040): No heartbeat from core client for 30 sec - exiting 02:41:19 (5040): No heartbeat from core client for 30 sec - exiting 02:41:20 (5040): No heartbeat from core client for 30 sec - exiting 02:41:21 (5040): No heartbeat from core client for 30 sec - exiting 02:41:22 (5040): No heartbeat from core client for 30 sec - exiting 02:41:23 (5040): No heartbeat from core client for 30 sec - exiting 02:41:24 (5040): No heartbeat from core client for 30 sec - exiting 02:41:25 (5040): No heartbeat from core client for 30 sec - exiting 02:41:26 (5040): No heartbeat from core client for 30 sec - exiting 02:41:27 (5040): No heartbeat from core client for 30 sec - exiting 02:41:28 (5040): No heartbeat from core client for 30 sec - exiting 02:41:30 (5040): No heartbeat from core client for 30 sec - exiting 02:41:31 (5040): No heartbeat from core client for 30 sec - exiting 02:41:32 (5040): No heartbeat from core client for 30 sec - exiting 02:41:33 (5040): No heartbeat from core client for 30 sec - exiting 02:41:34 (5040): No heartbeat from core client for 30 sec - exiting 02:41:35 (5040): No heartbeat from core client for 30 sec - exiting 02:41:36 (5040): No heartbeat from core client for 30 sec - exiting 02:41:37 (5040): No heartbeat from core client for 30 sec - exiting 02:41:38 (5040): No heartbeat from core client for 30 sec - exiting 02:41:39 (5040): No heartbeat from core client for 30 sec - exiting 02:41:40 (5040): No heartbeat from core client for 30 sec - exiting 02:41:42 (5040): No heartbeat from core client for 30 sec - exiting 02:41:43 (5040): No heartbeat from core client for 30 sec - exiting 02:41:44 (5040): No heartbeat from core client for 30 sec - exiting 02:41:45 (5040): No heartbeat from core client for 30 sec - exiting 02:42:19 (5040): No heartbeat from core client for 30 sec - exiting 02:42:20 (5040): No heartbeat from core client for 30 sec - exiting 02:42:21 (5040): No heartbeat from core client for 30 sec - exiting 02:42:22 (5040): No heartbeat from core client for 30 sec - exiting 02:42:23 (5040): No heartbeat from core client for 30 sec - exiting 02:42:24 (5040): No heartbeat from core client for 30 sec - exiting 02:42:25 (5040): No heartbeat from core client for 30 sec - exiting 02:42:26 (5040): No heartbeat from core client for 30 sec - exiting 02:42:27 (5040): No heartbeat from core client for 30 sec - exiting 02:42:29 (5040): No heartbeat from core client for 30 sec - exiting 02:42:30 (5040): No heartbeat from core client for 30 sec - exiting 02:42:31 (5040): No heartbeat from core client for 30 sec - exiting 02:42:32 (5040): No heartbeat from core client for 30 sec - exiting 02:42:33 (5040): No heartbeat from core client for 30 sec - exiting 02:42:34 (5040): No heartbeat from core client for 30 sec - exiting 02:42:35 (5040): No heartbeat from core client for 30 sec - exiting 02:42:36 (5040): No heartbeat from core client for 30 sec - exiting 02:42:37 (5040): No heartbeat from core client for 30 sec - exiting 02:42:38 (5040): No heartbeat from core client for 30 sec - exiting 02:42:39 (5040): No heartbeat from core client for 30 sec - exiting 02:42:41 (5040): No heartbeat from core client for 30 sec - exiting 02:42:42 (5040): No heartbeat from core client for 30 sec - exiting 02:42:43 (5040): No heartbeat from core client for 30 sec - exiting 02:42:44 (5040): No heartbeat from core client for 30 sec - exiting 02:42:45 (5040): No heartbeat from core client for 30 sec - exiting 02:42:46 (5040): No heartbeat from core client for 30 sec - exiting 02:42:47 (5040): No heartbeat from core client for 30 sec - exiting 02:42:48 (5040): No heartbeat from core client for 30 sec - exiting 02:42:49 (5040): No heartbeat from core client for 30 sec - exiting 02:42:50 (5040): No heartbeat from core client for 30 sec - exiting 02:42:51 (5040): No heartbeat from core client for 30 sec - exiting 02:42:53 (5040): No heartbeat from core client for 30 sec - exiting 02:42:54 (5040): No heartbeat from core client for 30 sec - exiting 02:42:55 (5040): No heartbeat from core client for 30 sec - exiting 02:42:56 (5040): No heartbeat from core client for 30 sec - exiting 02:42:57 (5040): No heartbeat from core client for 30 sec - exiting 02:42:58 (5040): No heartbeat from core client for 30 sec - exiting 02:42:59 (5040): No heartbeat from core client for 30 sec - exiting 02:43:00 (5040): No heartbeat from core client for 30 sec - exiting 02:43:01 (5040): No heartbeat from core client for 30 sec - exiting 02:43:02 (5040): No heartbeat from core client for 30 sec - exiting 02:43:03 (5040): No heartbeat from core client for 30 sec - exiting 02:43:05 (5040): No heartbeat from core client for 30 sec - exiting 02:43:06 (5040): No heartbeat from core client for 30 sec - exiting 02:43:07 (5040): No heartbeat from core client for 30 sec - exiting 02:43:08 (5040): No heartbeat from core client for 30 sec - exiting 02:43:42 (5040): No heartbeat from core client for 30 sec - exiting 02:43:43 (5040): No heartbeat from core client for 30 sec - exiting 02:43:44 (5040): No heartbeat from core client for 30 sec - exiting 02:43:45 (5040): No heartbeat from core client for 30 sec - exiting 02:43:46 (5040): No heartbeat from core client for 30 sec - exiting 02:43:47 (5040): No heartbeat from core client for 30 sec - exiting 02:43:48 (5040): No heartbeat from core client for 30 sec - exiting 02:43:49 (5040): No heartbeat from core client for 30 sec - exiting 02:43:50 (5040): No heartbeat from core client for 30 sec - exiting 02:43:52 (5040): No heartbeat from core client for 30 sec - exiting 02:43:53 (5040): No heartbeat from core client for 30 sec - exiting 02:43:54 (5040): No heartbeat from core client for 30 sec - exiting 02:43:55 (5040): No heartbeat from core client for 30 sec - exiting 02:43:56 (5040): No heartbeat from core client for 30 sec - exiting 02:43:57 (5040): No heartbeat from core client for 30 sec - exiting 02:43:58 (5040): No heartbeat from core client for 30 sec - exiting 02:43:59 (5040): No heartbeat from core client for 30 sec - exiting 02:44:00 (5040): No heartbeat from core client for 30 sec - exiting 02:44:01 (5040): No heartbeat from core client for 30 sec - exiting 02:44:02 (5040): No heartbeat from core client for 30 sec - exiting 02:44:04 (5040): No heartbeat from core client for 30 sec - exiting 02:44:05 (5040): No heartbeat from core client for 30 sec - exiting 02:44:06 (5040): No heartbeat from core client for 30 sec - exiting 02:44:07 (5040): No heartbeat from core client for 30 sec - exiting 02:44:08 (5040): No heartbeat from core client for 30 sec - exiting 02:44:09 (5040): No heartbeat from core client for 30 sec - exiting 02:44:10 (5040): No heartbeat from core client for 30 sec - exiting 02:44:11 (5040): No heartbeat from core client for 30 sec - exiting 02:44:45 (5040): No heartbeat from core client for 30 sec - exiting 02:44:46 (5040): No heartbeat from core client for 30 sec - exiting 02:44:47 (5040): No heartbeat from core client for 30 sec - exiting 02:44:48 (5040): No heartbeat from core client for 30 sec - exiting 02:44:49 (5040): No heartbeat from core client for 30 sec - exiting 02:44:50 (5040): No heartbeat from core client for 30 sec - exiting 02:44:52 (5040): No heartbeat from core client for 30 sec - exiting 02:44:53 (5040): No heartbeat from core client for 30 sec - exiting 02:44:54 (5040): No heartbeat from core client for 30 sec - exiting 02:44:55 (5040): No heartbeat from core client for 30 sec - exiting 02:44:56 (5040): No heartbeat from core client for 30 sec - exiting 02:44:57 (5040): No heartbeat from core client for 30 sec - exiting 02:44:58 (5040): No heartbeat from core client for 30 sec - exiting 02:44:59 (5040): No heartbeat from core client for 30 sec - exiting 02:45:00 (5040): No heartbeat from core client for 30 sec - exiting 02:45:01 (5040): No heartbeat from core client for 30 sec - exiting 02:45:02 (5040): No heartbeat from core client for 30 sec - exiting 02:45:04 (5040): No heartbeat from core client for 30 sec - exiting 02:45:05 (5040): No heartbeat from core client for 30 sec - exiting 02:45:06 (5040): No heartbeat from core client for 30 sec - exiting 02:45:07 (5040): No heartbeat from core client for 30 sec - exiting 02:45:08 (5040): No heartbeat from core client for 30 sec - exiting 02:45:09 (5040): No heartbeat from core client for 30 sec - exiting 02:45:10 (5040): No heartbeat from core client for 30 sec - exiting 02:45:11 (5040): No heartbeat from core client for 30 sec - exiting 02:45:12 (5040): No heartbeat from core client for 30 sec - exiting 02:45:13 (5040): No heartbeat from core client for 30 sec - exiting 02:45:14 (5040): No heartbeat from core client for 30 sec - exiting 02:45:48 (5040): No heartbeat from core client for 30 sec - exiting 02:45:49 (5040): No heartbeat from core client for 30 sec - exiting 02:45:51 (5040): No heartbeat from core client for 30 sec - exiting 02:45:52 (5040): No heartbeat from core client for 30 sec - exiting 02:45:53 (5040): No heartbeat from core client for 30 sec - exiting 02:45:54 (5040): No heartbeat from core client for 30 sec - exiting 02:45:55 (5040): No heartbeat from core client for 30 sec - exiting 02:45:56 (5040): No heartbeat from core client for 30 sec - exiting 02:45:57 (5040): No heartbeat from core client for 30 sec - exiting 02:45:58 (5040): No heartbeat from core client for 30 sec - exiting 02:45:59 (5040): No heartbeat from core client for 30 sec - exiting 02:46:00 (5040): No heartbeat from core client for 30 sec - exiting 02:46:01 (5040): No heartbeat from core client for 30 sec - exiting 02:46:03 (5040): No heartbeat from core client for 30 sec - exiting 02:46:04 (5040): No heartbeat from core client for 30 sec - exiting 02:46:05 (5040): No heartbeat from core client for 30 sec - exiting 02:46:06 (5040): No heartbeat from core client for 30 sec - exiting 02:46:07 (5040): No heartbeat from core client for 30 sec - exiting 02:46:08 (5040): No heartbeat from core client for 30 sec - exiting 02:46:09 (5040): No heartbeat from core client for 30 sec - exiting 02:46:10 (5040): No heartbeat from core client for 30 sec - exiting 02:46:11 (5040): No heartbeat from core client for 30 sec - exiting 02:46:12 (5040): No heartbeat from core client for 30 sec - exiting 02:46:13 (5040): No heartbeat from core client for 30 sec - exiting 02:46:15 (5040): No heartbeat from core client for 30 sec - exiting 02:46:16 (5040): No heartbeat from core client for 30 sec - exiting 02:46:17 (5040): No heartbeat from core client for 30 sec - exiting 02:46:18 (5040): No heartbeat from core client for 30 sec - exiting 02:46:19 (5040): No heartbeat from core client for 30 sec - exiting 02:46:20 (5040): No heartbeat from core client for 30 sec - exiting 02:46:21 (5040): No heartbeat from core client for 30 sec - exiting 02:46:22 (5040): No heartbeat from core client for 30 sec - exiting 02:46:23 (5040): No heartbeat from core client for 30 sec - exiting 02:46:24 (5040): No heartbeat from core client for 30 sec - exiting 02:46:25 (5040): No heartbeat from core client for 30 sec - exiting 02:46:27 (5040): No heartbeat from core client for 30 sec - exiting 02:46:28 (5040): No heartbeat from core client for 30 sec - exiting 02:46:29 (5040): No heartbeat from core client for 30 sec - exiting 02:46:30 (5040): No heartbeat from core client for 30 sec - exiting 02:46:31 (5040): No heartbeat from core client for 30 sec - exiting 02:46:32 (5040): No heartbeat from core client for 30 sec - exiting 02:46:33 (5040): No heartbeat from core client for 30 sec - exiting 02:46:34 (5040): No heartbeat from core client for 30 sec - exiting 02:46:35 (5040): No heartbeat from core client for 30 sec - exiting 02:46:36 (5040): No heartbeat from core client for 30 sec - exiting 02:46:37 (5040): No heartbeat from core client for 30 sec - exiting 02:46:39 (5040): No heartbeat from core client for 30 sec - exiting 02:46:40 (5040): No heartbeat from core client for 30 sec - exiting 02:46:41 (5040): No heartbeat from core client for 30 sec - exiting 02:46:42 (5040): No heartbeat from core client for 30 sec - exiting 02:46:43 (5040): No heartbeat from core client for 30 sec - exiting 02:46:44 (5040): No heartbeat from core client for 30 sec - exiting 02:46:45 (5040): No heartbeat from core client for 30 sec - exiting 02:46:46 (5040): No heartbeat from core client for 30 sec - exiting 02:47:20 (5040): No heartbeat from core client for 30 sec - exiting 02:47:21 (5040): No heartbeat from core client for 30 sec - exiting 02:47:22 (5040): No heartbeat from core client for 30 sec - exiting 02:47:23 (5040): No heartbeat from core client for 30 sec - exiting 02:47:24 (5040): No heartbeat from core client for 30 sec - exiting 02:47:26 (5040): No heartbeat from core client for 30 sec - exiting 02:47:27 (5040): No heartbeat from core client for 30 sec - exiting 02:47:28 (5040): No heartbeat from core client for 30 sec - exiting 02:47:29 (5040): No heartbeat from core client for 30 sec - exiting 02:47:30 (5040): No heartbeat from core client for 30 sec - exiting 02:47:31 (5040): No heartbeat from core client for 30 sec - exiting 02:47:32 (5040): No heartbeat from core client for 30 sec - exiting 02:47:33 (5040): No heartbeat from core client for 30 sec - exiting 02:47:34 (5040): No heartbeat from core client for 30 sec - exiting 02:47:35 (5040): No heartbeat from core client for 30 sec - exiting 02:47:36 (5040): No heartbeat from core client for 30 sec - exiting 02:47:38 (5040): No heartbeat from core client for 30 sec - exiting 02:47:39 (5040): No heartbeat from core client for 30 sec - exiting 02:47:40 (5040): No heartbeat from core client for 30 sec - exiting 02:47:41 (5040): No heartbeat from core client for 30 sec - exiting 02:47:42 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:21 (6716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:39:59 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:28 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Dec 2011 02:11:42	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	362,880	1,136,420	3.1317
11 Dec 2011 02:20:15	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	336,960	1,054,440	3.1293
10 Dec 2011 02:44:46	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	311,040	970,553	3.1203
09 Dec 2011 04:11:33	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	285,120	889,431	3.1195
08 Dec 2011 04:36:54	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	259,200	808,579	3.1195
07 Dec 2011 06:10:05	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	233,280	727,973	3.1206
06 Dec 2011 07:34:29	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	207,360	646,982	3.1201
05 Dec 2011 08:15:48	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	181,440	563,641	3.1065
04 Dec 2011 06:59:32	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	155,520	482,665	3.1036
03 Dec 2011 08:06:05	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	129,600	401,452	3.0976
02 Dec 2011 09:12:05	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	103,680	319,705	3.0836
01 Dec 2011 10:06:42	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	77,760	238,897	3.0722
30 Nov 2011 11:12:20	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	51,840	159,788	3.0823
29 Nov 2011 10:32:11	1122757	13664065	hadcm3n_u53j_1980_40_007539683_2	25,920	77,763	3.0001