Task 15273567

Name	hadcm3n_zkin_1880_40_008198354_0
Workunit	8353478
Created	12 Sep 2012, 17:42:18 UTC
Sent	12 Sep 2012, 17:49:35 UTC
Report deadline	13 Dec 2012, 1:16:46 UTC
Received	3 Nov 2012, 7:14:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1212349
Run time	2 days 4 hours 40 min 31 sec
CPU time	1 days 13 hours 26 min 9 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	1.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7044, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5300, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:56:57 (2160): No heartbeat from core client for 30 sec - exiting 08:56:59 (2160): No heartbeat from core client for 30 sec - exiting 08:57:00 (2160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6900, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4664, iMonCtr=1 Model crash detected, will try to restart... 21:33:23 (4732): No heartbeat from core client for 30 sec - exiting 21:33:24 (4732): No heartbeat from core client for 30 sec - exiting 21:33:25 (4732): No heartbeat from core client for 30 sec - exiting 21:33:26 (4732): No heartbeat from core client for 30 sec - exiting 21:33:27 (4732): No heartbeat from core client for 30 sec - exiting 21:33:28 (4732): No heartbeat from core client for 30 sec - exiting 21:33:29 (4732): No heartbeat from core client for 30 sec - exiting 21:33:31 (4732): No heartbeat from core client for 30 sec - exiting 21:33:32 (4732): No heartbeat from core client for 30 sec - exiting 21:33:33 (4732): No heartbeat from core client for 30 sec - exiting 21:33:34 (4732): No heartbeat from core client for 30 sec - exiting 21:33:35 (4732): No heartbeat from core client for 30 sec - exiting 21:33:36 (4732): No heartbeat from core client for 30 sec - exiting 21:33:37 (4732): No heartbeat from core client for 30 sec - exiting 21:33:38 (4732): No heartbeat from core client for 30 sec - exiting 21:33:39 (4732): No heartbeat from core client for 30 sec - exiting 21:33:40 (4732): No heartbeat from core client for 30 sec - exiting 21:33:41 (4732): No heartbeat from core client for 30 sec - exiting 21:33:43 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5180, iMonCtr=1 Model crash detected, will try to restart... 15:29:39 (1748): No heartbeat from core client for 30 sec - exiting 15:29:40 (1748): No heartbeat from core client for 30 sec - exiting 15:29:41 (1748): No heartbeat from core client for 30 sec - exiting 15:29:42 (1748): No heartbeat from core client for 30 sec - exiting 15:29:43 (1748): No heartbeat from core client for 30 sec - exiting 15:29:44 (1748): No heartbeat from core client for 30 sec - exiting 15:29:45 (1748): No heartbeat from core client for 30 sec - exiting 15:29:46 (1748): No heartbeat from core client for 30 sec - exiting 15:29:47 (1748): No heartbeat from core client for 30 sec - exiting 15:29:48 (1748): No heartbeat from core client for 30 sec - exiting 15:29:50 (1748): No heartbeat from core client for 30 sec - exiting 15:29:51 (1748): No heartbeat from core client for 30 sec - exiting 15:29:52 (1748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:52:20 (6040): No heartbeat from core client for 30 sec - exiting 20:52:21 (6040): No heartbeat from core client for 30 sec - exiting 20:52:22 (6040): No heartbeat from core client for 30 sec - exiting 20:52:23 (6040): No heartbeat from core client for 30 sec - exiting 20:52:24 (6040): No heartbeat from core client for 30 sec - exiting 20:52:25 (6040): No heartbeat from core client for 30 sec - exiting 20:52:26 (6040): No heartbeat from core client for 30 sec - exiting 20:52:27 (6040): No heartbeat from core client for 30 sec - exiting 20:52:29 (6040): No heartbeat from core client for 30 sec - exiting 20:52:30 (6040): No heartbeat from core client for 30 sec - exiting 20:52:31 (6040): No heartbeat from core client for 30 sec - exiting 20:52:32 (6040): No heartbeat from core client for 30 sec - exiting 20:52:33 (6040): No heartbeat from core client for 30 sec - exiting 20:52:34 (6040): No heartbeat from core client for 30 sec - exiting 20:52:35 (6040): No heartbeat from core client for 30 sec - exiting 20:52:36 (6040): No heartbeat from core client for 30 sec - exiting 20:52:37 (6040): No heartbeat from core client for 30 sec - exiting 20:52:38 (6040): No heartbeat from core client for 30 sec - exiting 20:52:39 (6040): No heartbeat from core client for 30 sec - exiting 20:52:41 (6040): No heartbeat from core client for 30 sec - exiting 20:52:42 (6040): No heartbeat from core client for 30 sec - exiting 20:52:43 (6040): No heartbeat from core client for 30 sec - exiting 20:52:44 (6040): No heartbeat from core client for 30 sec - exiting 20:52:45 (6040): No heartbeat from core client for 30 sec - exiting 20:52:46 (6040): No heartbeat from core client for 30 sec - exiting 20:52:47 (6040): No heartbeat from core client for 30 sec - exiting 20:52:48 (6040): No heartbeat from core client for 30 sec - exiting 20:52:49 (6040): No heartbeat from core client for 30 sec - exiting 20:52:50 (6040): No heartbeat from core client for 30 sec - exiting 20:52:52 (6040): No heartbeat from core client for 30 sec - exiting 20:52:53 (6040): No heartbeat from core client for 30 sec - exiting 20:52:54 (6040): No heartbeat from core client for 30 sec - exiting 20:52:55 (6040): No heartbeat from core client for 30 sec - exiting 20:52:56 (6040): No heartbeat from core client for 30 sec - exiting 20:52:57 (6040): No heartbeat from core client for 30 sec - exiting 20:52:58 (6040): No heartbeat from core client for 30 sec - exiting 20:52:59 (6040): No heartbeat from core client for 30 sec - exiting 20:53:00 (6040): No heartbeat from core client for 30 sec - exiting 20:53:01 (6040): No heartbeat from core client for 30 sec - exiting 20:53:02 (6040): No heartbeat from core client for 30 sec - exiting 20:53:04 (6040): No heartbeat from core client for 30 sec - exiting 20:53:05 (6040): No heartbeat from core client for 30 sec - exiting 20:53:06 (6040): No heartbeat from core client for 30 sec - exiting 20:53:07 (6040): No heartbeat from core client for 30 sec - exiting 20:53:08 (6040): No heartbeat from core client for 30 sec - exiting 20:53:09 (6040): No heartbeat from core client for 30 sec - exiting 20:53:10 (6040): No heartbeat from core client for 30 sec - exiting 20:53:11 (6040): No heartbeat from core client for 30 sec - exiting 20:53:12 (6040): No heartbeat from core client for 30 sec - exiting 20:53:13 (6040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:32 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:59:18 (3168): No heartbeat from core client for 30 sec - exiting 22:59:19 (3168): No heartbeat from core client for 30 sec - exiting 22:59:20 (3168): No heartbeat from core client for 30 sec - exiting 22:59:21 (3168): No heartbeat from core client for 30 sec - exiting 22:59:22 (3168): No heartbeat from core client for 30 sec - exiting 22:59:23 (3168): No heartbeat from core client for 30 sec - exiting 22:59:25 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:52:07 (5512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6536, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=740, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=740, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=740, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2792, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2792, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5052, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5052, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5052, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5616, iMonCtr=1 Model crash detected, will try to restart... 14:32:31 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:33:12 (6920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:46:17 (4460): No heartbeat from core client for 30 sec - exiting 10:46:18 (4460): No heartbeat from core client for 30 sec - exiting 10:46:19 (4460): No heartbeat from core client for 30 sec - exiting 10:46:20 (4460): No heartbeat from core client for 30 sec - exiting 10:46:21 (4460): No heartbeat from core client for 30 sec - exiting 10:46:22 (4460): No heartbeat from core client for 30 sec - exiting 10:46:23 (4460): No heartbeat from core client for 30 sec - exiting 10:46:24 (4460): No heartbeat from core client for 30 sec - exiting 10:46:25 (4460): No heartbeat from core client for 30 sec - exiting 10:46:04 (4460): No heartbeat from core client for 30 sec - exiting 10:46:05 (4460): No heartbeat from core client for 30 sec - exiting 10:46:06 (4460): No heartbeat from core client for 30 sec - exiting 10:46:07 (4460): No heartbeat from core client for 30 sec - exiting 10:46:08 (4460): No heartbeat from core client for 30 sec - exiting 10:46:09 (4460): No heartbeat from core client for 30 sec - exiting 10:46:10 (4460): No heartbeat from core client for 30 sec - exiting 10:46:11 (4460): No heartbeat from core client for 30 sec - exiting 10:46:13 (4460): No heartbeat from core client for 30 sec - exiting 10:46:14 (4460): No heartbeat from core client for 30 sec - exiting 10:46:15 (4460): No heartbeat from core client for 30 sec - exiting 10:46:16 (4460): No heartbeat from core client for 30 sec - exiting 10:46:17 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9356, iMonCtr=1 Model crash detected, will try to restart... 17:11:56 (6628): No heartbeat from core client for 30 sec - exiting 17:11:57 (6628): No heartbeat from core client for 30 sec - exiting 17:11:58 (6628): No heartbeat from core client for 30 sec - exiting 17:12:00 (6628): No heartbeat from core client for 30 sec - exiting 17:12:01 (6628): No heartbeat from core client for 30 sec - exiting 17:12:02 (6628): No heartbeat from core client for 30 sec - exiting 17:12:03 (6628): No heartbeat from core client for 30 sec - exiting 17:12:04 (6628): No heartbeat from core client for 30 sec - exiting 17:12:05 (6628): No heartbeat from core client for 30 sec - exiting 17:12:06 (6628): No heartbeat from core client for 30 sec - exiting 17:12:07 (6628): No heartbeat from core client for 30 sec - exiting 17:12:08 (6628): No heartbeat from core client for 30 sec - exiting 17:12:09 (6628): No heartbeat from core client for 30 sec - exiting 17:12:10 (6628): No heartbeat from core client for 30 sec - exiting 17:12:12 (6628): No heartbeat from core client for 30 sec - exiting 17:12:13 (6628): No heartbeat from core client for 30 sec - exiting 17:12:14 (6628): No heartbeat from core client for 30 sec - exiting 17:12:15 (6628): No heartbeat from core client for 30 sec - exiting 17:12:16 (6628): No heartbeat from core client for 30 sec - exiting 17:12:17 (6628): No heartbeat from core client for 30 sec - exiting 17:12:18 (6628): No heartbeat from core client for 30 sec - exiting 17:12:19 (6628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:29:25 (5532): No heartbeat from core client for 30 sec - exiting 15:29:27 (5532): No heartbeat from core client for 30 sec - exiting 15:29:28 (5532): No heartbeat from core client for 30 sec - exiting 15:29:29 (5532): No heartbeat from core client for 30 sec - exiting 15:29:30 (5532): No heartbeat from core client for 30 sec - exiting 15:29:31 (5532): No heartbeat from core client for 30 sec - exiting 15:29:32 (5532): No heartbeat from core client for 30 sec - exiting 15:29:33 (5532): No heartbeat from core client for 30 sec - exiting 15:29:34 (5532): No heartbeat from core client for 30 sec - exiting 15:29:35 (5532): No heartbeat from core client for 30 sec - exiting 15:29:36 (5532): No heartbeat from core client for 30 sec - exiting 15:29:37 (5532): No heartbeat from core client for 30 sec - exiting 15:29:39 (5532): No heartbeat from core client for 30 sec - exiting 15:29:40 (5532): No heartbeat from core client for 30 sec - exiting 15:29:41 (5532): No heartbeat from core client for 30 sec - exiting 15:29:42 (5532): No heartbeat from core client for 30 sec - exiting 15:29:43 (5532): No heartbeat from core client for 30 sec - exiting 15:29:44 (5532): No heartbeat from core client for 30 sec - exiting 15:29:45 (5532): No heartbeat from core client for 30 sec - exiting 15:29:46 (5532): No heartbeat from core client for 30 sec - exiting 15:29:47 (5532): No heartbeat from core client for 30 sec - exiting 15:29:48 (5532): No heartbeat from core client for 30 sec - exiting 15:29:49 (5532): No heartbeat from core client for 30 sec - exiting 15:29:51 (5532): No heartbeat from core client for 30 sec - exiting 15:29:52 (5532): No heartbeat from core client for 30 sec - exiting 15:29:53 (5532): No heartbeat from core client for 30 sec - exiting 15:29:54 (5532): No heartbeat from core client for 30 sec - exiting 15:29:55 (5532): No heartbeat from core client for 30 sec - exiting 15:29:56 (5532): No heartbeat from core client for 30 sec - exiting 15:29:57 (5532): No heartbeat from core client for 30 sec - exiting 15:29:58 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:47:04 (8688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:14:35 (5464): No heartbeat from core client for 30 sec - exiting 13:14:36 (5464): No heartbeat from core client for 30 sec - exiting 13:14:37 (5464): No heartbeat from core client for 30 sec - exiting 13:14:38 (5464): No heartbeat from core client for 30 sec - exiting 13:14:39 (5464): No heartbeat from core client for 30 sec - exiting 13:14:40 (5464): No heartbeat from core client for 30 sec - exiting 13:14:41 (5464): No heartbeat from core client for 30 sec - exiting 13:14:42 (5464): No heartbeat from core client for 30 sec - exiting 13:14:43 (5464): No heartbeat from core client for 30 sec - exiting 13:14:44 (5464): No heartbeat from core client for 30 sec - exiting 13:14:46 (5464): No heartbeat from core client for 30 sec - exiting 13:14:47 (5464): No heartbeat from core client for 30 sec - exiting 13:14:48 (5464): No heartbeat from core client for 30 sec - exiting 13:14:49 (5464): No heartbeat from core client for 30 sec - exiting 13:14:50 (5464): No heartbeat from core client for 30 sec - exiting 13:14:51 (5464): No heartbeat from core client for 30 sec - exiting 13:14:52 (5464): No heartbeat from core client for 30 sec - exiting 13:14:53 (5464): No heartbeat from core client for 30 sec - exiting 13:14:54 (5464): No heartbeat from core client for 30 sec - exiting 13:14:55 (5464): No heartbeat from core client for 30 sec - exiting 13:14:57 (5464): No heartbeat from core client for 30 sec - exiting 13:14:58 (5464): No heartbeat from core client for 30 sec - exiting 13:14:59 (5464): No heartbeat from core client for 30 sec - exiting 13:15:00 (5464): No heartbeat from core client for 30 sec - exiting 13:15:01 (5464): No heartbeat from core client for 30 sec - exiting 13:15:02 (5464): No heartbeat from core client for 30 sec - exiting 13:15:03 (5464): No heartbeat from core client for 30 sec - exiting 13:15:04 (5464): No heartbeat from core client for 30 sec - exiting 13:15:05 (5464): No heartbeat from core client for 30 sec - exiting 13:15:06 (5464): No heartbeat from core client for 30 sec - exiting 13:15:07 (5464): No heartbeat from core client for 30 sec - exiting 13:15:09 (5464): No heartbeat from core client for 30 sec - exiting 13:15:10 (5464): No heartbeat from core client for 30 sec - exiting 13:15:11 (5464): No heartbeat from core client for 30 sec - exiting 13:15:12 (5464): No heartbeat from core client for 30 sec - exiting 13:15:13 (5464): No heartbeat from core client for 30 sec - exiting 13:15:14 (5464): No heartbeat from core client for 30 sec - exiting 13:15:15 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:32:51 (5004): No heartbeat from core client for 30 sec - exiting 12:32:52 (5004): No heartbeat from core client for 30 sec - exiting 12:32:53 (5004): No heartbeat from core client for 30 sec - exiting 12:32:54 (5004): No heartbeat from core client for 30 sec - exiting 12:32:55 (5004): No heartbeat from core client for 30 sec - exiting 12:32:56 (5004): No heartbeat from core client for 30 sec - exiting 12:32:57 (5004): No heartbeat from core client for 30 sec - exiting 12:32:58 (5004): No heartbeat from core client for 30 sec - exiting 12:32:59 (5004): No heartbeat from core client for 30 sec - exiting 12:33:00 (5004): No heartbeat from core client for 30 sec - exiting 12:33:02 (5004): No heartbeat from core client for 30 sec - exiting 12:33:03 (5004): No heartbeat from core client for 30 sec - exiting 12:33:04 (5004): No heartbeat from core client for 30 sec - exiting 12:33:05 (5004): No heartbeat from core client for 30 sec - exiting 12:33:06 (5004): No heartbeat from core client for 30 sec - exiting 12:33:07 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:11:33 (6344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:47:41 (6456): No heartbeat from core client for 30 sec - exiting 13:47:42 (6456): No heartbeat from core client for 30 sec - exiting 13:47:43 (6456): No heartbeat from core client for 30 sec - exiting 13:47:44 (6456): No heartbeat from core client for 30 sec - exiting 13:47:46 (6456): No heartbeat from core client for 30 sec - exiting 13:47:47 (6456): No heartbeat from core client for 30 sec - exiting 13:47:48 (6456): No heartbeat from core client for 30 sec - exiting 13:47:49 (6456): No heartbeat from core client for 30 sec - exiting 13:47:50 (6456): No heartbeat from core client for 30 sec - exiting 13:47:51 (6456): No heartbeat from core client for 30 sec - exiting 13:47:52 (6456): No heartbeat from core client for 30 sec - exiting 13:47:53 (6456): No heartbeat from core client for 30 sec - exiting 13:47:54 (6456): No heartbeat from core client for 30 sec - exiting 13:47:55 (6456): No heartbeat from core client for 30 sec - exiting 13:47:57 (6456): No heartbeat from core client for 30 sec - exiting 13:47:58 (6456): No heartbeat from core client for 30 sec - exiting 13:47:59 (6456): No heartbeat from core client for 30 sec - exiting 13:48:00 (6456): No heartbeat from core client for 30 sec - exiting 13:48:01 (6456): No heartbeat from core client for 30 sec - exiting 13:48:02 (6456): No heartbeat from core client for 30 sec - exiting 13:48:03 (6456): No heartbeat from core client for 30 sec - exiting 13:48:04 (6456): No heartbeat from core client for 30 sec - exiting 13:48:05 (6456): No heartbeat from core client for 30 sec - exiting 13:48:06 (6456): No heartbeat from core client for 30 sec - exiting 13:48:07 (6456): No heartbeat from core client for 30 sec - exiting 13:48:09 (6456): No heartbeat from core client for 30 sec - exiting 13:48:10 (6456): No heartbeat from core client for 30 sec - exiting 13:48:11 (6456): No heartbeat from core client for 30 sec - exiting 13:48:12 (6456): No heartbeat from core client for 30 sec - exiting 13:48:13 (6456): No heartbeat from core client for 30 sec - exiting 13:48:14 (6456): No heartbeat from core client for 30 sec - exiting 13:48:15 (6456): No heartbeat from core client for 30 sec - exiting 13:48:16 (6456): No heartbeat from core client for 30 sec - exiting 13:48:17 (6456): No heartbeat from core client for 30 sec - exiting 13:48:18 (6456): No heartbeat from core client for 30 sec - exiting 13:48:19 (6456): No heartbeat from core client for 30 sec - exiting 13:48:21 (6456): No heartbeat from core client for 30 sec - exiting 13:48:22 (6456): No heartbeat from core client for 30 sec - exiting 13:48:23 (6456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:29:42 (6688): No heartbeat from core client for 30 sec - exiting 15:29:43 (6688): No heartbeat from core client for 30 sec - exiting 15:29:44 (6688): No heartbeat from core client for 30 sec - exiting 15:29:46 (6688): No heartbeat from core client for 30 sec - exiting 15:29:47 (6688): No heartbeat from core client for 30 sec - exiting 15:29:48 (6688): No heartbeat from core client for 30 sec - exiting 15:29:49 (6688): No heartbeat from core client for 30 sec - exiting 15:29:50 (6688): No heartbeat from core client for 30 sec - exiting 15:29:51 (6688): No heartbeat from core client for 30 sec - exiting 15:29:52 (6688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2012 19:31:39	1212349	15273567	hadcm3n_zkin_1880_40_008198354_0	25,920	76,097	2.9358