Task 12921554

Name	hadcm3n_o23q_1940_40_007265147_1
Workunit	7463387
Created	2 Jun 2011, 5:28:02 UTC
Sent	2 Jun 2011, 5:28:15 UTC
Report deadline	1 Sep 2011, 12:55:26 UTC
Received	21 Jun 2011, 16:15:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1103559
Run time	9 days 17 hours 14 min 5 sec
CPU time	7 days 3 hours 34 min 39 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	1.96 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:45:45 (2776): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5204, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5204, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5056, iMonCtr=1 Model crash detected, will try to restart... 08:49:26 (4768): No heartbeat from core client for 30 sec - exiting 08:49:27 (4768): No heartbeat from core client for 30 sec - exiting 08:49:28 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... 07:37:30 (5108): No heartbeat from core client for 30 sec - exiting 07:37:31 (5108): No heartbeat from core client for 30 sec - exiting 07:37:32 (5108): No heartbeat from core client for 30 sec - exiting 07:37:33 (5108): No heartbeat from core client for 30 sec - exiting 07:37:34 (5108): No heartbeat from core client for 30 sec - exiting 07:37:35 (5108): No heartbeat from core client for 30 sec - exiting 07:37:36 (5108): No heartbeat from core client for 30 sec - exiting 07:37:38 (5108): No heartbeat from core client for 30 sec - exiting 07:37:39 (5108): No heartbeat from core client for 30 sec - exiting 07:37:40 (5108): No heartbeat from core client for 30 sec - exiting 07:37:41 (5108): No heartbeat from core client for 30 sec - exiting 07:37:42 (5108): No heartbeat from core client for 30 sec - exiting 07:37:43 (5108): No heartbeat from core client for 30 sec - exiting 07:37:44 (5108): No heartbeat from core client for 30 sec - exiting 07:37:45 (5108): No heartbeat from core client for 30 sec - exiting 07:37:46 (5108): No heartbeat from core client for 30 sec - exiting 07:37:47 (5108): No heartbeat from core client for 30 sec - exiting 07:37:48 (5108): No heartbeat from core client for 30 sec - exiting 07:37:49 (5108): No heartbeat from core client for 30 sec - exiting 07:37:50 (5108): No heartbeat from core client for 30 sec - exiting 07:37:51 (5108): No heartbeat from core client for 30 sec - exiting 07:37:52 (5108): No heartbeat from core client for 30 sec - exiting 07:37:53 (5108): No heartbeat from core client for 30 sec - exiting 07:37:54 (5108): No heartbeat from core client for 30 sec - exiting 07:37:55 (5108): No heartbeat from core client for 30 sec - exiting 07:37:56 (5108): No heartbeat from core client for 30 sec - exiting 07:37:57 (5108): No heartbeat from core client for 30 sec - exiting 07:37:58 (5108): No heartbeat from core client for 30 sec - exiting 07:37:59 (5108): No heartbeat from core client for 30 sec - exiting 07:38:00 (5108): No heartbeat from core client for 30 sec - exiting 07:38:01 (5108): No heartbeat from core client for 30 sec - exiting 07:38:02 (5108): No heartbeat from core client for 30 sec - exiting 07:38:03 (5108): No heartbeat from core client for 30 sec - exiting 07:38:04 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6096, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:35:51 (7728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:23:45 (1200): No heartbeat from core client for 30 sec - exiting 00:23:46 (1200): No heartbeat from core client for 30 sec - exiting 00:23:47 (1200): No heartbeat from core client for 30 sec - exiting 00:23:48 (1200): No heartbeat from core client for 30 sec - exiting 00:23:49 (1200): No heartbeat from core client for 30 sec - exiting 00:23:50 (1200): No heartbeat from core client for 30 sec - exiting 00:23:51 (1200): No heartbeat from core client for 30 sec - exiting 00:23:52 (1200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:18:46 (5208): No heartbeat from core client for 30 sec - exiting 17:18:47 (5208): No heartbeat from core client for 30 sec - exiting 17:18:48 (5208): No heartbeat from core client for 30 sec - exiting 17:18:49 (5208): No heartbeat from core client for 30 sec - exiting 17:18:50 (5208): No heartbeat from core client for 30 sec - exiting 17:18:51 (5208): No heartbeat from core client for 30 sec - exiting 17:18:52 (5208): No heartbeat from core client for 30 sec - exiting 17:18:53 (5208): No heartbeat from core client for 30 sec - exiting 17:18:54 (5208): No heartbeat from core client for 30 sec - exiting 17:18:55 (5208): No heartbeat from core client for 30 sec - exiting 17:18:56 (5208): No heartbeat from core client for 30 sec - exiting 17:18:57 (5208): No heartbeat from core client for 30 sec - exiting 17:18:58 (5208): No heartbeat from core client for 30 sec - exiting 17:18:59 (5208): No heartbeat from core client for 30 sec - exiting 17:19:00 (5208): No heartbeat from core client for 30 sec - exiting 17:19:01 (5208): No heartbeat from core client for 30 sec - exiting 17:19:02 (5208): No heartbeat from core client for 30 sec - exiting 17:19:03 (5208): No heartbeat from core client for 30 sec - exiting 17:19:04 (5208): No heartbeat from core client for 30 sec - exiting 17:19:05 (5208): No heartbeat from core client for 30 sec - exiting 17:19:06 (5208): No heartbeat from core client for 30 sec - exiting 17:19:07 (5208): No heartbeat from core client for 30 sec - exiting 17:19:08 (5208): No heartbeat from core client for 30 sec - exiting 17:19:09 (5208): No heartbeat from core client for 30 sec - exiting 17:19:10 (5208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:04:21 (5772): No heartbeat from core client for 30 sec - exiting 06:04:22 (5772): No heartbeat from core client for 30 sec - exiting 06:04:23 (5772): No heartbeat from core client for 30 sec - exiting 06:04:24 (5772): No heartbeat from core client for 30 sec - exiting 06:04:25 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:34:32 (7968): No heartbeat from core client for 30 sec - exiting 18:34:33 (7968): No heartbeat from core client for 30 sec - exiting 18:34:34 (7968): No heartbeat from core client for 30 sec - exiting 18:34:35 (7968): No heartbeat from core client for 30 sec - exiting 18:34:36 (7968): No heartbeat from core client for 30 sec - exiting 18:34:37 (7968): No heartbeat from core client for 30 sec - exiting 18:34:38 (7968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:30:46 (5272): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 05:13:39 (3980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:49 (1048): No heartbeat from core client for 30 sec - exiting 05:35:50 (1048): No heartbeat from core client for 30 sec - exiting 05:35:51 (1048): No heartbeat from core client for 30 sec - exiting 05:35:52 (1048): No heartbeat from core client for 30 sec - exiting 05:35:53 (1048): No heartbeat from core client for 30 sec - exiting 05:35:54 (1048): No heartbeat from core client for 30 sec - exiting 05:35:55 (1048): No heartbeat from core client for 30 sec - exiting 05:35:56 (1048): No heartbeat from core client for 30 sec - exiting 05:35:57 (1048): No heartbeat from core client for 30 sec - exiting 05:35:58 (1048): No heartbeat from core client for 30 sec - exiting 05:35:59 (1048): No heartbeat from core client for 30 sec - exiting 05:36:00 (1048): No heartbeat from core client for 30 sec - exiting 05:36:01 (1048): No heartbeat from core client for 30 sec - exiting 05:36:02 (1048): No heartbeat from core client for 30 sec - exiting 05:36:03 (1048): No heartbeat from core client for 30 sec - exiting 05:36:04 (1048): No heartbeat from core client for 30 sec - exiting 05:36:05 (1048): No heartbeat from core client for 30 sec - exiting 05:36:06 (1048): No heartbeat from core client for 30 sec - exiting 05:36:07 (1048): No heartbeat from core client for 30 sec - exiting 05:36:08 (1048): No heartbeat from core client for 30 sec - exiting 05:36:09 (1048): No heartbeat from core client for 30 sec - exiting 05:36:10 (1048): No heartbeat from core client for 30 sec - exiting 05:36:11 (1048): No heartbeat from core client for 30 sec - exiting 05:36:12 (1048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:22:12 (3016): No heartbeat from core client for 30 sec - exiting 17:22:13 (3016): No heartbeat from core client for 30 sec - exiting 17:22:14 (3016): No heartbeat from core client for 30 sec - exiting 17:22:15 (3016): No heartbeat from core client for 30 sec - exiting 17:22:16 (3016): No heartbeat from core client for 30 sec - exiting 17:22:17 (3016): No heartbeat from core client for 30 sec - exiting 17:22:18 (3016): No heartbeat from core client for 30 sec - exiting 17:22:19 (3016): No heartbeat from core client for 30 sec - exiting 17:22:20 (3016): No heartbeat from core client for 30 sec - exiting 17:22:21 (3016): No heartbeat from core client for 30 sec - exiting 17:22:22 (3016): No heartbeat from core client for 30 sec - exiting 17:22:23 (3016): No heartbeat from core client for 30 sec - exiting 17:22:24 (3016): No heartbeat from core client for 30 sec - exiting 17:22:25 (3016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:42:45 (5412): No heartbeat from core client for 30 sec - exiting 19:42:46 (5412): No heartbeat from core client for 30 sec - exiting 19:49:21 (5568): No heartbeat from core client for 30 sec - exiting 19:49:22 (5568): No heartbeat from core client for 30 sec - exiting 19:49:23 (5568): No heartbeat from core client for 30 sec - exiting 19:49:24 (5568): No heartbeat from core client for 30 sec - exiting 19:49:25 (5568): No heartbeat from core client for 30 sec - exiting 19:49:26 (5568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:14:32 (7540): No heartbeat from core client for 30 sec - exiting 21:14:33 (7540): No heartbeat from core client for 30 sec - exiting 21:14:34 (7540): No heartbeat from core client for 30 sec - exiting 21:14:35 (7540): No heartbeat from core client for 30 sec - exiting 21:14:36 (7540): No heartbeat from core client for 30 sec - exiting 21:14:37 (7540): No heartbeat from core client for 30 sec - exiting 21:14:38 (7540): No heartbeat from core client for 30 sec - exiting 21:14:39 (7540): No heartbeat from core client for 30 sec - exiting 21:14:40 (7540): No heartbeat from core client for 30 sec - exiting 21:14:41 (7540): No heartbeat from core client for 30 sec - exiting 21:14:42 (7540): No heartbeat from core client for 30 sec - exiting 21:14:43 (7540): No heartbeat from core client for 30 sec - exiting 21:14:44 (7540): No heartbeat from core client for 30 sec - exiting 21:14:45 (7540): No heartbeat from core client for 30 sec - exiting 21:14:46 (7540): No heartbeat from core client for 30 sec - exiting 21:14:47 (7540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:40:19 (4924): No heartbeat from core client for 30 sec - exiting 08:40:21 (4924): No heartbeat from core client for 30 sec - exiting 08:40:22 (4924): No heartbeat from core client for 30 sec - exiting 08:40:23 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:24 (4924): No heartbeat from core client for 30 sec - exiting C08:57:21 (7224): No heartbeat from core client for 30 sec - exiting 08:57:22 (7224): No heartbeat from core client for 30 sec - exiting 08:57:23 (7224): No heartbeat from core client for 30 sec - exiting 08:57:24 (7224): No heartbeat from core client for 30 sec - exiting 08:57:25 (7224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:09 (6348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:34 (7552): No heartbeat from core client for 30 sec - exiting 09:47:35 (7552): No heartbeat from core client for 30 sec - exiting 09:47:36 (7552): No heartbeat from core client for 30 sec - exiting 09:47:37 (7552): No heartbeat from core client for 30 sec - exiting 09:47:38 (7552): No heartbeat from core client for 30 sec - exiting 09:47:39 (7552): No heartbeat from core client for 30 sec - exiting 09:47:40 (7552): No heartbeat from core client for 30 sec - exiting 09:47:41 (7552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:32:56 (6400): No heartbeat from core client for 30 sec - exiting 12:32:57 (6400): No heartbeat from core client for 30 sec - exiting 12:32:58 (6400): No heartbeat from core client for 30 sec - exiting 12:32:59 (6400): No heartbeat from core client for 30 sec - exiting 12:33:00 (6400): No heartbeat from core client for 30 sec - exiting 12:33:01 (6400): No heartbeat from core client for 30 sec - exiting 12:33:02 (6400): No heartbeat from core client for 30 sec - exiting 12:33:03 (6400): No heartbeat from core client for 30 sec - exiting 12:33:04 (6400): No heartbeat from core client for 30 sec - exiting 12:33:05 (6400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:33:42 (7496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:54:25 (7524): No heartbeat from core client for 30 sec - exiting 17:54:26 (7524): No heartbeat from core client for 30 sec - exiting 17:54:27 (7524): No heartbeat from core client for 30 sec - exiting 17:54:28 (7524): No heartbeat from core client for 30 sec - exiting 17:54:29 (7524): No heartbeat from core client for 30 sec - exiting 17:54:30 (7524): No heartbeat from core client for 30 sec - exiting 17:54:31 (7524): No heartbeat from core client for 30 sec - exiting 17:54:32 (7524): No heartbeat from core client for 30 sec - exiting 17:54:33 (7524): No heartbeat from core client for 30 sec - exiting 17:54:34 (7524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:48:14 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6128, iMonCtr=1 Model crash detected, will try to restart... 16:18:06 (4684): No heartbeat from core client for 30 sec - exiting 16:18:07 (4684): No heartbeat from core client for 30 sec - exiting 16:18:08 (4684): No heartbeat from core client for 30 sec - exiting 16:18:09 (4684): No heartbeat from core client for 30 sec - exiting 16:18:10 (4684): No heartbeat from core client for 30 sec - exiting 16:18:11 (4684): No heartbeat from core client for 30 sec - exiting 16:18:12 (4684): No heartbeat from core client for 30 sec - exiting 16:18:13 (4684): No heartbeat from core client for 30 sec - exiting 16:18:14 (4684): No heartbeat from core client for 30 sec - exiting 16:18:15 (4684): No heartbeat from core client for 30 sec - exiting 16:18:16 (4684): No heartbeat from core client for 30 sec - exiting 16:18:17 (4684): No heartbeat from core client for 30 sec - exiting 16:18:18 (4684): No heartbeat from core client for 30 sec - exiting 16:18:19 (4684): No heartbeat from core client for 30 sec - exiting 16:18:20 (4684): No heartbeat from core client for 30 sec - exiting 16:18:21 (4684): No heartbeat from core client for 30 sec - exiting 16:18:22 (4684): No heartbeat from core client for 30 sec - exiting 16:18:23 (4684): No heartbeat from core client for 30 sec - exiting 16:18:24 (4684): No heartbeat from core client for 30 sec - exiting 16:18:25 (4684): No heartbeat from core client for 30 sec - exiting 16:18:26 (4684): No heartbeat from core client for 30 sec - exiting 16:18:27 (4684): No heartbeat from core client for 30 sec - exiting 16:18:28 (4684): No heartbeat from core client for 30 sec - exiting 16:18:29 (4684): No heartbeat from core client for 30 sec - exiting 16:18:30 (4684): No heartbeat from core client for 30 sec - exiting 16:18:31 (4684): No heartbeat from core client for 30 sec - exiting 16:18:32 (4684): No heartbeat from core client for 30 sec - exiting 16:18:33 (4684): No heartbeat from core client for 30 sec - exiting 16:18:34 (4684): No heartbeat from core client for 30 sec - exiting 16:18:35 (4684): No heartbeat from core client for 30 sec - exiting 16:18:36 (4684): No heartbeat from core client for 30 sec - exiting 16:18:37 (4684): No heartbeat from core client for 30 sec - exiting 16:18:38 (4684): No heartbeat from core client for 30 sec - exiting 16:18:39 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:17:16 (6028): No heartbeat from core client for 30 sec - exiting 06:17:17 (6028): No heartbeat from core client for 30 sec - exiting 06:17:18 (6028): No heartbeat from core client for 30 sec - exiting 06:17:19 (6028): No heartbeat from core client for 30 sec - exiting 06:17:20 (6028): No heartbeat from core client for 30 sec - exiting 06:17:21 (6028): No heartbeat from core client for 30 sec - exiting 06:17:22 (6028): No heartbeat from core client for 30 sec - exiting 06:17:23 (6028): No heartbeat from core client for 30 sec - exiting 06:17:24 (6028): No heartbeat from core client for 30 sec - exiting 06:17:25 (6028): No heartbeat from core client for 30 sec - exiting 06:17:26 (6028): No heartbeat from core client for 30 sec - exiting 06:17:27 (6028): No heartbeat from core client for 30 sec - exiting 06:17:28 (6028): No heartbeat from core client for 30 sec - exiting 06:17:29 (6028): No heartbeat from core client for 30 sec - exiting 06:17:30 (6028): No heartbeat from core client for 30 sec - exiting 06:17:31 (6028): No heartbeat from core client for 30 sec - exiting 06:17:32 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:07:12 (4024): No heartbeat from core client for 30 sec - exiting 17:07:13 (4024): No heartbeat from core client for 30 sec - exiting 17:07:14 (4024): No heartbeat from core client for 30 sec - exiting 17:07:15 (4024): No heartbeat from core client for 30 sec - exiting 17:07:16 (4024): No heartbeat from core client for 30 sec - exiting 17:07:17 (4024): No heartbeat from core client for 30 sec - exiting 17:07:18 (4024): No heartbeat from core client for 30 sec - exiting 17:07:19 (4024): No heartbeat from core client for 30 sec - exiting 17:07:20 (4024): No heartbeat from core client for 30 sec - exiting 17:07:21 (4024): No heartbeat from core client for 30 sec - exiting 17:07:22 (4024): No heartbeat from core client for 30 sec - exiting 17:07:23 (4024): No heartbeat from core client for 30 sec - exiting 17:07:24 (4024): No heartbeat from core client for 30 sec - exiting 17:07:25 (4024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4784, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish 17:11:00 (2712): No heartbeat from core client for 30 sec - exiting 17:11:01 (2712): No heartbeat from core client for 30 sec - exiting 17:11:02 (2712): No heartbeat from core client for 30 sec - exiting 17:11:03 (2712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6060, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6060, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6060, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jun 2011 17:20:39	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	311,040	581,614	1.8699
19 Jun 2011 23:05:06	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	285,120	530,446	1.8604
19 Jun 2011 22:04:06	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	259,200	479,415	1.8496
17 Jun 2011 11:41:33	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	233,280	428,887	1.8385
15 Jun 2011 08:14:38	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	207,360	377,592	1.8209
14 Jun 2011 07:41:07	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	181,440	327,107	1.8028
12 Jun 2011 02:20:50	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	155,520	275,903	1.7741
11 Jun 2011 01:41:44	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	129,600	224,240	1.7302
10 Jun 2011 05:23:30	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	103,680	173,883	1.6771
09 Jun 2011 07:25:10	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	77,760	153,529	1.9744
08 Jun 2011 12:28:27	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	51,840	102,599	1.9791
07 Jun 2011 21:00:56	1103559	12921554	hadcm3n_o23q_1940_40_007265147_1	25,920	51,375	1.9821