Task 12988002

Name	hadcm3n_s1zk_1940_40_007298872_1
Workunit	7496296
Created	20 Jun 2011, 15:39:07 UTC
Sent	20 Jun 2011, 15:39:12 UTC
Report deadline	19 Sep 2011, 23:06:23 UTC
Received	4 Jul 2011, 1:11:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1154239
Run time	6 days 17 hours 22 min 11 sec
CPU time	4 days 22 hours 25 min 35 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	1.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:58:19 (2516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:42:41 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:38:21 (4856): No heartbeat from core client for 30 sec - exiting 12:38:22 (4856): No heartbeat from core client for 30 sec - exiting 12:38:23 (4856): No heartbeat from core client for 30 sec - exiting 12:38:24 (4856): No heartbeat from core client for 30 sec - exiting 12:38:25 (4856): No heartbeat from core client for 30 sec - exiting 12:38:26 (4856): No heartbeat from core client for 30 sec - exiting 12:38:27 (4856): No heartbeat from core client for 30 sec - exiting 12:38:28 (4856): No heartbeat from core client for 30 sec - exiting 12:38:29 (4856): No heartbeat from core client for 30 sec - exiting 12:38:31 (4856): No heartbeat from core client for 30 sec - exiting 12:38:32 (4856): No heartbeat from core client for 30 sec - exiting 12:38:33 (4856): No heartbeat from core client for 30 sec - exiting 12:38:34 (4856): No heartbeat from core client for 30 sec - exiting 12:38:35 (4856): No heartbeat from core client for 30 sec - exiting 12:38:36 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:38:37 (4856): No heartbeat from core client for 30 sec - exiting 12:38:38 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:45:19 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on s1zkko.dae14b0 CPDN Monitor - Quit request from BOINC... 09:34:01 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:42:24 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:49:02 (5264): No heartbeat from core client for 30 sec - exiting 09:49:03 (5264): No heartbeat from core client for 30 sec - exiting 09:49:04 (5264): No heartbeat from core client for 30 sec - exiting 09:49:05 (5264): No heartbeat from core client for 30 sec - exiting 09:49:06 (5264): No heartbeat from core client for 30 sec - exiting 09:49:07 (5264): No heartbeat from core client for 30 sec - exiting 09:49:08 (5264): No heartbeat from core client for 30 sec - exiting 09:49:09 (5264): No heartbeat from core client for 30 sec - exiting 09:49:10 (5264): No heartbeat from core client for 30 sec - exiting 09:49:11 (5264): No heartbeat from core client for 30 sec - exiting 09:49:12 (5264): No heartbeat from core client for 30 sec - exiting 09:49:13 (5264): No heartbeat from core client for 30 sec - exiting 09:49:14 (5264): No heartbeat from core client for 30 sec - exiting 09:49:15 (5264): No heartbeat from core client for 30 sec - exiting 09:49:17 (5264): No heartbeat from core client for 30 sec - exiting 09:49:18 (5264): No heartbeat from core client for 30 sec - exiting 09:49:19 (5264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:53:59 (4864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:58:24 (5380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:25 (5380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:35:13 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:44:01 (6132): No heartbeat from core client for 30 sec - exiting 13:44:02 (6132): No heartbeat from core client for 30 sec - exiting 13:44:03 (6132): No heartbeat from core client for 30 sec - exiting 13:44:04 (6132): No heartbeat from core client for 30 sec - exiting 13:44:05 (6132): No heartbeat from core client for 30 sec - exiting 13:44:10 (6132): No heartbeat from core client for 30 sec - exiting 13:44:11 (6132): No heartbeat from core client for 30 sec - exiting 13:44:13 (6132): No heartbeat from core client for 30 sec - exiting 13:44:18 (6132): No heartbeat from core client for 30 sec - exiting 13:44:21 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:22 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:51:28 (3456): No heartbeat from core client for 30 sec - exiting 13:51:29 (3456): No heartbeat from core client for 30 sec - exiting 13:51:30 (3456): No heartbeat from core client for 30 sec - exiting 13:51:31 (3456): No heartbeat from core client for 30 sec - exiting 13:51:32 (3456): No heartbeat from core client for 30 sec - exiting 13:51:33 (3456): No heartbeat from core client for 30 sec - exiting 13:51:34 (3456): No heartbeat from core client for 30 sec - exiting 13:51:35 (3456): No heartbeat from core client for 30 sec - exiting 13:51:36 (3456): No heartbeat from core client for 30 sec - exiting 13:51:37 (3456): No heartbeat from core client for 30 sec - exiting 13:51:38 (3456): No heartbeat from core client for 30 sec - exiting 13:51:39 (3456): No heartbeat from core client for 30 sec - exiting 13:51:40 (3456): No heartbeat from core client for 30 sec - exiting 13:51:41 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:59:10 (3248): No heartbeat from core client for 30 sec - exiting 13:59:11 (3248): No heartbeat from core client for 30 sec - exiting 13:59:12 (3248): No heartbeat from core client for 30 sec - exiting 13:59:13 (3248): No heartbeat from core client for 30 sec - exiting 13:59:14 (3248): No heartbeat from core client for 30 sec - exiting 13:59:15 (3248): No heartbeat from core client for 30 sec - exiting 13:59:17 (3248): No heartbeat from core client for 30 sec - exiting 13:59:18 (3248): No heartbeat from core client for 30 sec - exiting 13:59:19 (3248): No heartbeat from core client for 30 sec - exiting 13:59:20 (3248): No heartbeat from core client for 30 sec - exiting 13:59:21 (3248): No heartbeat from core client for 30 sec - exiting 13:59:22 (3248): No heartbeat from core client for 30 sec - exiting 13:59:23 (3248): No heartbeat from core client for 30 sec - exiting 13:59:24 (3248): No heartbeat from core client for 30 sec - exiting 13:59:25 (3248): No heartbeat from core client for 30 sec - exiting 13:59:26 (3248): No heartbeat from core client for 30 sec - exiting 13:59:27 (3248): No heartbeat from core client for 30 sec - exiting 13:59:28 (3248): No heartbeat from core client for 30 sec - exiting 13:59:29 (3248): No heartbeat from core client for 30 sec - exiting 13:59:30 (3248): No heartbeat from core client for 30 sec - exiting 13:59:31 (3248): No heartbeat from core client for 30 sec - exiting 13:59:32 (3248): No heartbeat from core client for 30 sec - exiting 13:59:33 (3248): No heartbeat from core client for 30 sec - exiting 13:59:34 (3248): No heartbeat from core client for 30 sec - exiting 13:59:35 (3248): No heartbeat from core client for 30 sec - exiting 13:59:36 (3248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:37 (3248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:09:37 (2864): No heartbeat from core client for 30 sec - exiting 15:09:38 (2864): No heartbeat from core client for 30 sec - exiting 15:09:39 (2864): No heartbeat from core client for 30 sec - exiting 15:09:40 (2864): No heartbeat from core client for 30 sec - exiting 15:09:41 (2864): No heartbeat from core client for 30 sec - exiting 15:09:42 (2864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:50:52 (4760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on s1zkko.dae29t0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:23:59 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:27:30 (7980): No heartbeat from core client for 30 sec - exiting 13:27:31 (7980): No heartbeat from core client for 30 sec - exiting 13:27:32 (7980): No heartbeat from core client for 30 sec - exiting 13:27:33 (7980): No heartbeat from core client for 30 sec - exiting 13:27:34 (7980): No heartbeat from core client for 30 sec - exiting 13:27:35 (7980): No heartbeat from core client for 30 sec - exiting 13:27:36 (7980): No heartbeat from core client for 30 sec - exiting 13:27:37 (7980): No heartbeat from core client for 30 sec - exiting 13:27:38 (7980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:07:03 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:00 (7548): No heartbeat from core client for 30 sec - exiting 14:08:02 (7548): No heartbeat from core client for 30 sec - exiting 14:08:03 (7548): No heartbeat from core client for 30 sec - exiting 14:08:05 (7548): No heartbeat from core client for 30 sec - exiting 14:08:07 (7548): No heartbeat from core client for 30 sec - exiting 14:08:08 (7548): No heartbeat from core client for 30 sec - exiting 14:08:09 (7548): No heartbeat from core client for 30 sec - exiting 14:08:10 (7548): No heartbeat from core client for 30 sec - exiting 14:08:11 (7548): No heartbeat from core client for 30 sec - exiting 14:08:12 (7548): No heartbeat from core client for 30 sec - exiting 14:08:13 (7548): No heartbeat from core client for 30 sec - exiting 14:08:14 (7548): No heartbeat from core client for 30 sec - exiting 14:08:15 (7548): No heartbeat from core client for 30 sec - exiting 14:08:16 (7548): No heartbeat from core client for 30 sec - exiting 14:08:17 (7548): No heartbeat from core client for 30 sec - exiting 14:08:18 (7548): No heartbeat from core client for 30 sec - exiting 14:08:19 (7548): No heartbeat from core client for 30 sec - exiting 14:08:20 (7548): No heartbeat from core client for 30 sec - exiting 14:08:21 (7548): No heartbeat from core client for 30 sec - exiting 14:08:22 (7548): No heartbeat from core client for 30 sec - exiting 14:08:23 (7548): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. 14:08:24 (7548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11020, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 14:12:58 (3788): No heartbeat from core client for 30 sec - exiting 14:12:59 (3788): No heartbeat from core client for 30 sec - exiting 14:13:00 (3788): No heartbeat from core client for 30 sec - exiting 14:13:02 (3788): No heartbeat from core client for 30 sec - exiting 14:13:03 (3788): No heartbeat from core client for 30 sec - exiting 14:13:04 (3788): No heartbeat from core client for 30 sec - exiting 14:13:05 (3788): No heartbeat from core client for 30 sec - exiting 14:13:06 (3788): No heartbeat from core client for 30 sec - exiting 14:13:07 (3788): No heartbeat from core client for 30 sec - exiting 14:13:08 (3788): No heartbeat from core client for 30 sec - exiting 14:13:09 (3788): No heartbeat from core client for 30 sec - exiting 14:13:10 (3788): No heartbeat from core client for 30 sec - exiting 14:13:11 (3788): No heartbeat from core client for 30 sec - exiting 14:13:12 (3788): No heartbeat from core client for 30 sec - exiting 14:13:13 (3788): No heartbeat from core client for 30 sec - exiting 14:13:14 (3788): No heartbeat from core client for 30 sec - exiting 14:13:15 (3788): No heartbeat from core client for 30 sec - exiting 14:13:16 (3788): No heartbeat from core client for 30 sec - exiting 14:13:17 (3788): No heartbeat from core client for 30 sec - exiting 14:13:18 (3788): No heartbeat from core client for 30 sec - exiting 14:13:19 (3788): No heartbeat from core client for 30 sec - exiting 14:13:20 (3788): No heartbeat from core client for 30 sec - exiting 14:13:21 (3788): No heartbeat from core client for 30 sec - exiting 14:13:22 (3788): No heartbeat from core client for 30 sec - exiting 14:13:23 (3788): No heartbeat from core client for 30 sec - exiting 14:13:24 (3788): No heartbeat from core client for 30 sec - exiting 14:13:25 (3788): No heartbeat from core client for 30 sec - exiting 14:13:26 (3788): No heartbeat from core client for 30 sec - exiting 14:13:27 (3788): No heartbeat from core client for 30 sec - exiting 14:13:28 (3788): No heartbeat from core client for 30 sec - exiting 14:13:29 (3788): No heartbeat from core client for 30 sec - exiting 14:13:30 (3788): No heartbeat from core client for 30 sec - exiting 14:13:31 (3788): No heartbeat from core client for 30 sec - exiting 14:13:32 (3788): No heartbeat from core client for 30 sec - exiting 14:13:33 (3788): No heartbeat from core client for 30 sec - exiting 14:13:34 (3788): No heartbeat from core client for 30 sec - exiting 14:13:35 (3788): No heartbeat from core client for 30 sec - exiting 14:13:36 (3788): No heartbeat from core client for 30 sec - exiting 14:13:37 (3788): No heartbeat from core client for 30 sec - exiting 14:13:38 (3788): No heartbeat from core client for 30 sec - exiting 14:13:39 (3788): No heartbeat from core client for 30 sec - exiting 14:13:40 (3788): No heartbeat from core client for 30 sec - exiting 14:13:41 (3788): No heartbeat from core client for 30 sec - exiting 14:13:42 (3788): No heartbeat from core client for 30 sec - exiting 14:13:43 (3788): No heartbeat from core client for 30 sec - exiting 14:13:44 (3788): No heartbeat from core client for 30 sec - exiting 14:13:45 (3788): No heartbeat from core client for 30 sec - exiting 14:13:46 (3788): No heartbeat from core client for 30 sec - exiting 14:13:47 (3788): No heartbeat from core client for 30 sec - exiting 14:13:48 (3788): No heartbeat from core client for 30 sec - exiting 14:13:49 (3788): No heartbeat from core client for 30 sec - exiting 14:13:50 (3788): No heartbeat from core client for 30 sec - exiting 14:13:51 (3788): No heartbeat from core client for 30 sec - exiting 14:13:52 (3788): No heartbeat from core client for 30 sec - exiting 14:13:53 (3788): No heartbeat from core client for 30 sec - exiting 14:13:54 (3788): No heartbeat from core client for 30 sec - exiting 14:13:55 (3788): No heartbeat from core client for 30 sec - exiting 14:13:56 (3788): No heartbeat from core client for 30 sec - exiting 14:13:57 (3788): No heartbeat from core client for 30 sec - exiting 14:13:58 (3788): No heartbeat from core client for 30 sec - exiting 14:13:59 (3788): No heartbeat from core client for 30 sec - exiting 14:14:00 (3788): No heartbeat from core client for 30 sec - exiting 14:14:01 (3788): No heartbeat from core client for 30 sec - exiting 14:14:02 (3788): No heartbeat from core client for 30 sec - exiting 14:14:03 (3788): No heartbeat from core client for 30 sec - exiting 14:14:04 (3788): No heartbeat from core client for 30 sec - exiting 14:14:05 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:21:00 (12012): No heartbeat from core client for 30 sec - exiting 14:21:05 (12012): No heartbeat from core client for 30 sec - exiting 14:21:08 (12012): No heartbeat from core client for 30 sec - exiting 14:21:10 (12012): No heartbeat from core client for 30 sec - exiting 14:21:12 (12012): No heartbeat from core client for 30 sec - exiting 14:21:14 (12012): No heartbeat from core client for 30 sec - exiting 14:21:15 (12012): No heartbeat from core client for 30 sec - exiting 14:21:17 (12012): No heartbeat from core client for 30 sec - exiting 14:21:19 (12012): No heartbeat from core client for 30 sec - exiting 14:21:20 (12012): No heartbeat from core client for 30 sec - exiting 14:21:22 (12012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:21:50 (12012):CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:33:40 (8640): No heartbeat from core client for 30 sec - exiting 14:33:41 (8640): No heartbeat from core client for 30 sec - exiting 14:33:42 (8640): No heartbeat from core client for 30 sec - exiting 14:33:43 (8640): No heartbeat from core client for 30 sec - exiting 14:33:44 (8640): No heartbeat from core client for 30 sec - exiting 14:33:45 (8640): No heartbeat from core client for 30 sec - exiting 14:33:46 (8640): No heartbeat from core client for 30 sec - exiting 14:33:47 (8640): No heartbeat from core client for 30 sec - exiting 14:33:48 (8640): No heartbeat from core client for 30 sec - exiting 14:33:49 (8640): No heartbeat from core client for 30 sec - exiting 14:33:50 (8640): No heartbeat from core client for 30 sec - exiting 14:33:51 (8640): No heartbeat from core client for 30 sec - exiting 14:33:52 (8640): No heartbeat from core client for 30 sec - exiting 14:33:53 (8640): No heartbeat from core client for 30 sec - exiting 14:33:54 (8640): No heartbeat from core client for 30 sec - exiting 14:33:55 (8640): No heartbeat from core client for 30 sec - exiting 14:33:56 (8640): No heartbeat from core client for 30 sec - exiting 14:33:57 (8640): No heartbeat from core client for 30 sec - exiting 14:33:58 (8640): No heartbeat from core client for 30 sec - exiting 14:33:59 (8640): No heartbeat from core client for 30 sec - exiting 14:34:00 (8640): No heartbeat from core client for 30 sec - exiting 14:34:01 (8640): No heartbeat from core client for 30 sec - exiting 14:34:02 (8640): No heartbeat from core client for 30 sec - exiting 14:34:03 (8640): No heartbeat from core client for 30 sec - exiting 14:34:04 (8640): No heartbeat from core client for 30 sec - exiting 14:34:05 (8640): No heartbeat from core client for 30 sec - exiting 14:34:06 (8640): No heartbeat from core client for 30 sec - exiting 14:34:07 (8640): No heartbeat from core client for 30 sec - exiting 14:34:08 (8640): No heartbeat from core client for 30 sec - exiting 14:34:09 (8640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:34:10 (8640): No heartbeat from core client for 30 sec - exiting 14:34:11 (8640): No heartbeat from core client for 30 sec - exiting 14:34:12 (8640): No heartbeat from core client for 30 sec - exiting 14:34:13 (8640): No heartbeat from core client for 30 sec - exiting 14:34:14 (8640): No heartbeat from core client for 30 sec - exiting 14:34:15 (8640): No heartbeat from core client for 30 sec - exiting 14:34:16 (8640): No heartbeat from core client for 30 sec - exiting 14:34:17 (8640): No heartbeat from core client for 30 sec - exiting 14:34:18 (8640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:40:29 (11296): No heartbeat from core client for 30 sec - exiting 14:40:31 (11296): No heartbeat from core client for 30 sec - exiting 14:40:32 (11296): No heartbeat from core client for 30 sec - exiting 14:40:33 (11296): No heartbeat from core client for 30 sec - exiting 14:40:34 (11296): No heartbeat from core client for 30 sec - exiting 14:40:35 (11296): No heartbeat from core client for 30 sec - exiting 14:40:36 (11296): No heartbeat from core client for 30 sec - exiting 14:40:37 (11296): No heartbeat from core client for 30 sec - exiting 14:40:38 (11296): No heartbeat from core client for 30 sec - exiting 14:40:39 (11296): No heartbeat from core client for 30 sec - exiting 14:40:40 (11296): No heartbeat from core client for 30 sec - exiting 14:40:41 (11296): No heartbeat from core client for 30 sec - exiting 14:40:42 (11296): No heartbeat from core client for 30 sec - exiting 14:40:43 (11296): No heartbeat from core client for 30 sec - exiting 14:40:44 (11296): No heartbeat from core client for 30 sec - exiting 14:40:45 (11296): No heartbeat from core client for 30 sec - exiting 14:40:46 (11296): No heartbeat from core client for 30 sec - exiting 14:40:47 (11296): No heartbeat from core client for 30 sec - exiting 14:40:48 (11296): No heartbeat from core client for 30 sec - exiting 14:40:49 (11296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:44:35 (3644): No heartbeat from core client for 30 sec - exiting 14:44:36 (3644): No heartbeat from core client for 30 sec - exiting 14:44:37 (3644): No heartbeat from core client for 30 sec - exiting 14:44:38 (3644): No heartbeat from core client for 30 sec - exiting 14:44:39 (3644): No heartbeat from core client for 30 sec - exiting 14:44:40 (3644): No heartbeat from core client for 30 sec - exiting 14:44:41 (3644): No heartbeat from core client for 30 sec - exiting 14:44:42 (3644): No heartbeat from core client for 30 sec - exiting 14:44:43 (3644): No heartbeat from core client for 30 sec - exiting 14:44:44 (3644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:05:05 (5556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:51:04 (5368): No heartbeat from core client for 30 sec - exiting 12:51:05 (5368): No heartbeat from core client for 30 sec - exiting 12:51:06 (5368): No heartbeat from core client for 30 sec - exiting 12:51:07 (5368): No heartbeat from core client for 30 sec - exiting 12:51:08 (5368): No heartbeat from core client for 30 sec - exiting 12:51:09 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:58:21 (3520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:43:17 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:43:18 (4392): No heartbeat from core client for 30 sec - exiting 12:43:19 (4392): No heartbeat from core client for 30 sec - exiting 12:43:20 (4392): No heartbeat from core client for 30 sec - exiting 12:43:21 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:12:01 (7852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:41:36 (6276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:26:27 (6376): No heartbeat from core client for 30 sec - exiting 12:26:29 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:30 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:33:48 (7052): No heartbeat from core client for 30 sec - exiting 14:33:49 (7052): No heartbeat from core client for 30 sec - exiting 14:33:50 (7052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:02:57 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:02:58 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 16:17:56 (7168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jul 2011 01:12:58	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	233,280	484,411	2.0765
04 Jul 2011 01:12:58	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	207,360	429,414	2.0709
30 Jun 2011 13:12:51	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	181,440	375,156	2.0677
29 Jun 2011 14:57:05	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	155,520	321,698	2.0685
28 Jun 2011 17:56:51	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	129,600	271,544	2.0952
27 Jun 2011 11:04:39	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	103,680	217,283	2.0957
27 Jun 2011 01:26:50	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	77,760	162,403	2.0885
26 Jun 2011 02:39:20	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	51,840	108,576	2.0944
24 Jun 2011 13:13:46	1154239	12988002	hadcm3n_s1zk_1940_40_007298872_1	25,920	53,982	2.0826