Task 13148819

Name	hadcm3n_y9fg_1900_40_007345078_2
Workunit	7542508
Created	22 Jul 2011, 12:15:46 UTC
Sent	22 Jul 2011, 12:16:40 UTC
Report deadline	21 Oct 2011, 19:43:51 UTC
Received	10 Aug 2011, 10:17:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1283959
Run time	9 days 1 hours 25 min 51 sec
CPU time	6 days 18 hours 34 min 35 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	2.46 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.13.1</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:09:55 (4808): No heartbeat from core client for 30 sec - exiting 22:09:56 (4808): No heartbeat from core client for 30 sec - exiting 22:09:57 (4808): No heartbeat from core client for 30 sec - exiting 22:09:58 (4808): No heartbeat from core client for 30 sec - exiting 22:09:59 (4808): No heartbeat from core client for 30 sec - exiting 22:10:00 (4808): No heartbeat from core client for 30 sec - exiting 22:10:01 (4808): No heartbeat from core client for 30 sec - exiting 22:10:02 (4808): No heartbeat from core client for 30 sec - exiting 22:10:03 (4808): No heartbeat from core client for 30 sec - exiting 22:10:04 (4808): No heartbeat from core client for 30 sec - exiting 22:10:05 (4808): No heartbeat from core client for 30 sec - exiting 22:10:07 (4808): No heartbeat from core client for 30 sec - exiting 22:10:08 (4808): No heartbeat from core client for 30 sec - exiting 22:10:09 (4808): No heartbeat from core client for 30 sec - exiting 22:10:10 (4808): No heartbeat from core client for 30 sec - exiting 22:10:11 (4808): No heartbeat from core client for 30 sec - exiting 22:10:12 (4808): No heartbeat from core client for 30 sec - exiting 22:10:13 (4808): No heartbeat from core client for 30 sec - exiting 22:10:14 (4808): No heartbeat from core client for 30 sec - exiting 22:10:15 (4808): No heartbeat from core client for 30 sec - exiting 22:10:16 (4808): No heartbeat from core client for 30 sec - exiting 22:10:17 (4808): No heartbeat from core client for 30 sec - exiting 22:10:18 (4808): No heartbeat from core client for 30 sec - exiting 22:10:19 (4808): No heartbeat from core client for 30 sec - exiting 22:10:21 (4808): No heartbeat from core client for 30 sec - exiting 22:10:22 (4808): No heartbeat from core client for 30 sec - exiting 22:10:23 (4808): No heartbeat from core client for 30 sec - exiting 22:10:24 (4808): No heartbeat from core client for 30 sec - exiting 22:10:25 (4808): No heartbeat from core client for 30 sec - exiting 22:10:26 (4808): No heartbeat from core client for 30 sec - exiting 22:10:27 (4808): No heartbeat from core client for 30 sec - exiting 22:10:28 (4808): No heartbeat from core client for 30 sec - exiting 22:10:29 (4808): No heartbeat from core client for 30 sec - exiting 22:10:30 (4808): No heartbeat from core client for 30 sec - exiting 22:10:31 (4808): No heartbeat from core client for 30 sec - exiting 22:10:32 (4808): No heartbeat from core client for 30 sec - exiting 22:10:33 (4808): No heartbeat from core client for 30 sec - exiting 22:10:34 (4808): No heartbeat from core client for 30 sec - exiting 22:10:35 (4808): No heartbeat from core client for 30 sec - exiting 22:10:36 (4808): No heartbeat from core client for 30 sec - exiting 22:10:37 (4808): No heartbeat from core client for 30 sec - exiting 22:10:38 (4808): No heartbeat from core client for 30 sec - exiting 22:10:39 (4808): No heartbeat from core client for 30 sec - exiting 22:10:40 (4808): No heartbeat from core client for 30 sec - exiting 22:10:41 (4808): No heartbeat from core client for 30 sec - exiting 22:10:42 (4808): No heartbeat from core client for 30 sec - exiting 22:10:43 (4808): No heartbeat from core client for 30 sec - exiting 22:10:44 (4808): No heartbeat from core client for 30 sec - exiting 22:10:45 (4808): No heartbeat from core client for 30 sec - exiting 22:10:46 (4808): No heartbeat from core client for 30 sec - exiting 22:10:47 (4808): No heartbeat from core client for 30 sec - exiting 22:10:48 (4808): No heartbeat from core client for 30 sec - exiting 22:10:50 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:51 (4808): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3908, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 20:16:25 (2764): No heartbeat from core client for 30 sec - exiting 20:16:27 (2764): No heartbeat from core client for 30 sec - exiting 20:16:28 (2764): No heartbeat from core client for 30 sec - exiting 20:16:29 (2764): No heartbeat from core client for 30 sec - exiting 20:16:30 (2764): No heartbeat from core client for 30 sec - exiting 20:16:31 (2764): No heartbeat from core client for 30 sec - exiting 20:16:32 (2764): No heartbeat from core client for 30 sec - exiting 20:16:33 (2764): No heartbeat from core client for 30 sec - exiting 20:16:34 (2764): No heartbeat from core client for 30 sec - exiting 20:16:35 (2764): No heartbeat from core client for 30 sec - exiting 20:16:36 (2764): No heartbeat from core client for 30 sec - exiting 20:16:37 (2764): No heartbeat from core client for 30 sec - exiting 20:16:38 (2764): No heartbeat from core client for 30 sec - exiting 20:16:39 (2764): No heartbeat from core client for 30 sec - exiting 20:16:40 (2764): No heartbeat from core client for 30 sec - exiting 20:16:41 (2764): No heartbeat from core client for 30 sec - exiting 20:16:42 (2764): No heartbeat from core client for 30 sec - exiting 20:16:43 (2764): No heartbeat from core client for 30 sec - exiting 20:16:44 (2764): No heartbeat from core client for 30 sec - exiting 20:16:45 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:46 (2764): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:44:23 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:00:20 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 17:05:45 (4168): No heartbeat from core client for 30 sec - exiting 17:05:46 (4168): No heartbeat from core client for 30 sec - exiting 17:05:47 (4168): No heartbeat from core client for 30 sec - exiting 17:05:48 (4168): No heartbeat from core client for 30 sec - exiting 17:05:49 (4168): No heartbeat from core client for 30 sec - exiting 17:05:50 (4168): No heartbeat from core client for 30 sec - exiting 17:05:51 (4168): No heartbeat from core client for 30 sec - exiting 17:05:52 (4168): No heartbeat from core client for 30 sec - exiting 17:05:53 (4168): No heartbeat from core client for 30 sec - exiting 17:05:54 (4168): No heartbeat from core client for 30 sec - exiting 17:05:55 (4168): No heartbeat from core client for 30 sec - exiting 17:05:56 (4168): No heartbeat from core client for 30 sec - exiting 17:05:57 (4168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:10:17 (3512): No heartbeat from core client for 30 sec - exiting 17:10:18 (3512): No heartbeat from core client for 30 sec - exiting 17:10:19 (3512): No heartbeat from core client for 30 sec - exiting 17:10:20 (3512): No heartbeat from core client for 30 sec - exiting 17:10:21 (3512): No heartbeat from core client for 30 sec - exiting 17:10:22 (3512): No heartbeat from core client for 30 sec - exiting 17:10:23 (3512): No heartbeat from core client for 30 sec - exiting 17:10:24 (3512): No heartbeat from core client for 30 sec - exiting 17:10:25 (3512): No heartbeat from core client for 30 sec - exiting 17:10:26 (3512): No heartbeat from core client for 30 sec - exiting 17:10:28 (3512): No heartbeat from core client for 30 sec - exiting 17:10:29 (3512): No heartbeat from core client for 30 sec - exiting 17:10:30 (3512): No heartbeat from core client for 30 sec - exiting 17:10:31 (3512): No heartbeat from core client for 30 sec - exiting 17:10:32 (3512): No heartbeat from core client for 30 sec - exiting 17:10:33 (3512): No heartbeat from core client for 30 sec - exiting 17:10:34 (3512): No heartbeat from core client for 30 sec - exiting 17:10:35 (3512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:16:58 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:29:23 (5052): No heartbeat from core client for 30 sec - exiting 08:29:24 (5052): No heartbeat from core client for 30 sec - exiting 08:29:25 (5052): No heartbeat from core client for 30 sec - exiting 08:29:26 (5052): No heartbeat from core client for 30 sec - exiting 08:29:27 (5052): No heartbeat from core client for 30 sec - exiting 08:29:28 (5052): No heartbeat from core client for 30 sec - exiting 08:29:29 (5052): No heartbeat from core client for 30 sec - exiting 08:29:30 (5052): No heartbeat from core client for 30 sec - exiting 08:29:31 (5052): No heartbeat from core client for 30 sec - exiting 08:29:32 (5052): No heartbeat from core client for 30 sec - exiting 08:29:33 (5052): No heartbeat from core client for 30 sec - exiting 08:29:34 (5052): No heartbeat from core client for 30 sec - exiting 08:29:35 (5052): No heartbeat from core client for 30 sec - exiting 08:29:37 (5052): No heartbeat from core client for 30 sec - exiting 08:29:38 (5052): No heartbeat from core client for 30 sec - exiting 08:29:39 (5052): No heartbeat from core client for 30 sec - exiting 08:29:40 (5052): No heartbeat from core client for 30 sec - exiting 08:29:41 (5052): No heartbeat from core client for 30 sec - exiting 08:29:42 (5052): No heartbeat from core client for 30 sec - exiting 08:29:43 (5052): No heartbeat from core client for 30 sec - exiting 08:29:44 (5052): No heartbeat from core client for 30 sec - exiting 08:29:45 (5052): No heartbeat from core client for 30 sec - exiting 08:29:46 (5052): No heartbeat from core client for 30 sec - exiting 08:29:47 (5052): No heartbeat from core client for 30 sec - exiting 08:29:48 (5052): No heartbeat from core client for 30 sec - exiting 08:29:49 (5052): No heartbeat from core client for 30 sec - exiting 08:29:50 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:32 (3140): No heartbeat from core client for 30 sec - exiting 14:40:33 (3140): No heartbeat from core client for 30 sec - exiting 14:40:34 (3140): No heartbeat from core client for 30 sec - exiting 14:40:35 (3140): No heartbeat from core client for 30 sec - exiting 14:40:36 (3140): No heartbeat from core client for 30 sec - exiting 14:40:37 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2564, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4100, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:48:45 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1032, iMonCtr=1 Model crash detected, will try to restart... 01:30:41 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:27 (3012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:52:34 (4576): No heartbeat from core client for 30 sec - exiting 15:52:35 (4576): No heartbeat from core client for 30 sec - exiting 15:52:36 (4576): No heartbeat from core client for 30 sec - exiting 15:52:37 (4576): No heartbeat from core client for 30 sec - exiting 15:52:38 (4576): No heartbeat from core client for 30 sec - exiting 15:52:39 (4576): No heartbeat from core client for 30 sec - exiting 15:52:40 (4576): No heartbeat from core client for 30 sec - exiting 15:52:41 (4576): No heartbeat from core client for 30 sec - exiting 15:52:42 (4576): No heartbeat from core client for 30 sec - exiting 15:52:43 (4576): No heartbeat from core client for 30 sec - exiting 15:52:45 (4576): No heartbeat from core client for 30 sec - exiting 15:52:46 (4576): No heartbeat from core client for 30 sec - exiting 15:52:47 (4576): No heartbeat from core client for 30 sec - exiting 15:52:48 (4576): No heartbeat from core client for 30 sec - exiting 15:52:49 (4576): No heartbeat from core client for 30 sec - exiting 15:52:50 (4576): No heartbeat from core client for 30 sec - exiting 15:52:51 (4576): No heartbeat from core client for 30 sec - exiting 15:52:52 (4576): No heartbeat from core client for 30 sec - exiting 15:52:53 (4576): No heartbeat from core client for 30 sec - exiting 15:52:54 (4576): No heartbeat from core client for 30 sec - exiting 15:52:55 (4576): No heartbeat from core client for 30 sec - exiting 15:52:56 (4576): No heartbeat from core client for 30 sec - exiting 15:52:57 (4576): No heartbeat from core client for 30 sec - exiting 15:52:58 (4576): No heartbeat from core client for 30 sec - exiting 15:52:59 (4576): No heartbeat from core client for 30 sec - exiting 15:53:00 (4576): No heartbeat from core client for 30 sec - exiting 15:53:01 (4576): No heartbeat from core client for 30 sec - exiting 15:53:02 (4576): No heartbeat from core client for 30 sec - exiting 15:53:03 (4576): No heartbeat from core client for 30 sec - exiting 15:53:04 (4576): No heartbeat from core client for 30 sec - exiting 15:53:05 (4576): No heartbeat from core client for 30 sec - exiting 15:53:06 (4576): No heartbeat from core client for 30 sec - exiting 15:53:07 (4576): No heartbeat from core client for 30 sec - exiting 15:53:08 (4576): No heartbeat from core client for 30 sec - exiting 15:53:09 (4576): No heartbeat from core client for 30 sec - exiting 15:53:10 (4576): No heartbeat from core client for 30 sec - exiting 15:53:11 (4576): No heartbeat from core client for 30 sec - exiting 15:53:12 (4576): No heartbeat from core client for 30 sec - exiting 15:53:13 (4576): No heartbeat from core client for 30 sec - exiting 15:53:14 (4576): No heartbeat from core client for 30 sec - exiting 15:53:16 (4576): No heartbeat from core client for 30 sec - exiting 15:53:17 (4576): No heartbeat from core client for 30 sec - exiting 15:53:18 (4576): No heartbeat from core client for 30 sec - exiting 15:53:19 (4576): No heartbeat from core client for 30 sec - exiting 15:53:20 (4576): No heartbeat from core client for 30 sec - exiting 15:53:21 (4576): No heartbeat from core client for 30 sec - exiting 15:53:22 (4576): No heartbeat from core client for 30 sec - exiting 15:53:23 (4576): No heartbeat from core client for 30 sec - exiting 15:53:24 (4576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:56:18 (3844): No heartbeat from core client for 30 sec - exiting 20:56:19 (3844): No heartbeat from core client for 30 sec - exiting 20:56:20 (3844): No heartbeat from core client for 30 sec - exiting 20:56:21 (3844): No heartbeat from core client for 30 sec - exiting 20:56:22 (3844): No heartbeat from core client for 30 sec - exiting 20:56:23 (3844): No heartbeat from core client for 30 sec - exiting 20:56:24 (3844): No heartbeat from core client for 30 sec - exiting 20:56:25 (3844): No heartbeat from core client for 30 sec - exiting 20:56:26 (3844): No heartbeat from core client for 30 sec - exiting 20:56:27 (3844): No heartbeat from core client for 30 sec - exiting 20:56:28 (3844): No heartbeat from core client for 30 sec - exiting 20:56:29 (3844): No heartbeat from core client for 30 sec - exiting 20:56:30 (3844): No heartbeat from core client for 30 sec - exiting 20:56:31 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:56:32 (3844): No heartbeat from core client for 30 sec - exiting 20:56:33 (3844): No heartbeat from core client for 30 sec - exiting 20:56:34 (3844): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13844, iMonCtr=1 Model crash detected, will try to restart... 20:25:05 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5040, iMonCtr=1 Model crash detected, will try to restart... 14:21:02 (4020): No heartbeat from core client for 30 sec - exiting 14:21:03 (4020): No heartbeat from core client for 30 sec - exiting 14:21:04 (4020): No heartbeat from core client for 30 sec - exiting 14:21:05 (4020): No heartbeat from core client for 30 sec - exiting 14:21:06 (4020): No heartbeat from core client for 30 sec - exiting 14:21:07 (4020): No heartbeat from core client for 30 sec - exiting 14:21:08 (4020): No heartbeat from core client for 30 sec - exiting 14:21:09 (4020): No heartbeat from core client for 30 sec - exiting 14:21:10 (4020): No heartbeat from core client for 30 sec - exiting 14:21:11 (4020): No heartbeat from core client for 30 sec - exiting 14:21:12 (4020): No heartbeat from core client for 30 sec - exiting 14:21:13 (4020): No heartbeat from core client for 30 sec - exiting 14:21:14 (4020): No heartbeat from core client for 30 sec - exiting 14:21:15 (4020): No heartbeat from core client for 30 sec - exiting 14:21:16 (4020): No heartbeat from core client for 30 sec - exiting 14:21:17 (4020): No heartbeat from core client for 30 sec - exiting 14:21:18 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3020, iMonCtr=1 Model crash detected, will try to restart... 22:51:10 (2724): No heartbeat from core client for 30 sec - exiting 22:51:12 (2724): No heartbeat from core client for 30 sec - exiting 22:51:14 (2724): No heartbeat from core client for 30 sec - exiting 22:51:15 (2724): No heartbeat from core client for 30 sec - exiting 22:51:16 (2724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:02:00 (3196): No heartbeat from core client for 30 sec - exiting 23:02:01 (3196): No heartbeat from core client for 30 sec - exiting 23:02:02 (3196): No heartbeat from core client for 30 sec - exiting 23:02:03 (3196): No heartbeat from core client for 30 sec - exiting 23:02:04 (3196): No heartbeat from core client for 30 sec - exiting 23:02:05 (3196): No heartbeat from core client for 30 sec - exiting 23:02:06 (3196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1080, iMonCtr=1 Model crash detected, will try to restart... 08:55:36 (3688): No heartbeat from core client for 30 sec - exiting 08:55:37 (3688): No heartbeat from core client for 30 sec - exiting 08:55:38 (3688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Aug 2011 01:42:33	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	285,120	578,911	2.0304
08 Aug 2011 03:04:17	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	259,200	528,845	2.0403
07 Aug 2011 07:56:50	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	233,280	473,444	2.0295
06 Aug 2011 11:42:59	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	207,360	420,302	2.0269
05 Aug 2011 02:54:09	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	181,440	368,741	2.0323
02 Aug 2011 23:48:49	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	155,520	314,738	2.0238
02 Aug 2011 03:31:03	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	129,600	262,887	2.0284
30 Jul 2011 07:05:25	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	103,680	209,391	2.0196
28 Jul 2011 12:23:27	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	77,760	157,260	2.0224
27 Jul 2011 13:14:06	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	51,840	101,675	1.9613
25 Jul 2011 22:55:19	956831	13148819	hadcm3n_y9fg_1900_40_007345078_2	25,920	51,221	1.9761