Task 15483836

Name	hadcm3n_za7y_1880_40_008200822_4
Workunit	8355946
Created	19 Dec 2012, 1:41:52 UTC
Sent	19 Dec 2012, 1:42:09 UTC
Report deadline	20 Mar 2013, 9:09:20 UTC
Received	1 Feb 2013, 4:33:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	892963
Run time	26 days 15 hours 1 min 6 sec
CPU time	20 days 1 hours 37 min 28 sec
Validate state	Invalid
Credit	9,020.16
Device peak FLOPS	2.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on za7yko.da82ae0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on za7yko.da859l0 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:03:06 (5976): No heartbeat from core client for 30 sec - exiting 09:03:07 (5976): No heartbeat from core client for 30 sec - exiting 09:03:09 (5976): No heartbeat from core client for 30 sec - exiting 09:03:10 (5976): No heartbeat from core client for 30 sec - exiting 09:03:11 (5976): No heartbeat from core client for 30 sec - exiting 09:03:12 (5976): No heartbeat from core client for 30 sec - exiting 09:03:13 (5976): No heartbeat from core client for 30 sec - exiting 09:03:14 (5976): No heartbeat from core client for 30 sec - exiting 09:03:15 (5976): No heartbeat from core client for 30 sec - exiting 09:03:16 (5976): No heartbeat from core client for 30 sec - exiting 09:03:17 (5976): No heartbeat from core client for 30 sec - exiting 09:03:18 (5976): No heartbeat from core client for 30 sec - exiting 09:03:19 (5976): No heartbeat from core client for 30 sec - exiting 09:03:21 (5976): No heartbeat from core client for 30 sec - exiting 09:03:22 (5976): No heartbeat from core client for 30 sec - exiting 09:03:23 (5976): No heartbeat from core client for 30 sec - exiting 09:03:24 (5976): No heartbeat from core client for 30 sec - exiting 09:03:25 (5976): No heartbeat from core client for 30 sec - exiting 09:03:26 (5976): No heartbeat from core client for 30 sec - exiting 09:03:27 (5976): No heartbeat from core client for 30 sec - exiting 09:03:28 (5976): No heartbeat from core client for 30 sec - exiting 09:03:29 (5976): No heartbeat from core client for 30 sec - exiting 09:03:30 (5976): No heartbeat from core client for 30 sec - exiting 09:03:32 (5976): No heartbeat from core client for 30 sec - exiting 09:03:33 (5976): No heartbeat from core client for 30 sec - exiting 09:03:34 (5976): No heartbeat from core client for 30 sec - exiting 09:03:35 (5976): No heartbeat from core client for 30 sec - exiting 09:03:36 (5976): No heartbeat from core client for 30 sec - exiting 09:03:37 (5976): No heartbeat from core client for 30 sec - exiting 09:03:38 (5976): No heartbeat from core client for 30 sec - exiting 09:03:39 (5976): No heartbeat from core client for 30 sec - exiting 09:03:40 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 23:18:56 (4672): No heartbeat from core client for 30 sec - exiting 23:18:57 (4672): No heartbeat from core client for 30 sec - exiting 23:18:58 (4672): No heartbeat from core client for 30 sec - exiting 23:18:59 (4672): No heartbeat from core client for 30 sec - exiting 23:19:00 (4672): No heartbeat from core client for 30 sec - exiting 23:19:01 (4672): No heartbeat from core client for 30 sec - exiting 23:19:02 (4672): No heartbeat from core client for 30 sec - exiting 23:19:03 (4672): No heartbeat from core client for 30 sec - exiting 23:19:04 (4672): No heartbeat from core client for 30 sec - exiting 23:19:06 (4672): No heartbeat from core client for 30 sec - exiting 23:19:07 (4672): No heartbeat from core client for 30 sec - exiting 23:19:08 (4672): No heartbeat from core client for 30 sec - exiting 23:19:09 (4672): No heartbeat from core client for 30 sec - exiting 23:19:10 (4672): No heartbeat from core client for 30 sec - exiting 23:19:15 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:16 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:47:42 (3824): No heartbeat from core client for 30 sec - exiting 23:47:43 (3824): No heartbeat from core client for 30 sec - exiting 23:47:44 (3824): No heartbeat from core client for 30 sec - exiting 23:47:45 (3824): No heartbeat from core client for 30 sec - exiting 23:47:46 (3824): No heartbeat from core client for 30 sec - exiting 23:47:48 (3824): No heartbeat from core client for 30 sec - exiting 23:47:49 (3824): No heartbeat from core client for 30 sec - exiting 23:47:50 (3824): No heartbeat from core client for 30 sec - exiting 23:47:51 (3824): No heartbeat from core client for 30 sec - exiting 23:47:52 (3824): No heartbeat from core client for 30 sec - exiting 23:47:53 (3824): No heartbeat from core client for 30 sec - exiting 23:47:54 (3824): No heartbeat from core client for 30 sec - exiting 23:47:55 (3824): No heartbeat from core client for 30 sec - exiting 23:47:56 (3824): No heartbeat from core client for 30 sec - exiting 23:47:57 (3824): No heartbeat from core client for 30 sec - exiting 23:47:59 (3824): No heartbeat from core client for 30 sec - exiting 23:48:00 (3824): No heartbeat from core client for 30 sec - exiting 23:48:01 (3824): No heartbeat from core client for 30 sec - exiting 23:48:02 (3824): No heartbeat from core client for 30 sec - exiting 23:48:03 (3824): No heartbeat from core client for 30 sec - exiting 23:48:04 (3824): No heartbeat from core client for 30 sec - exiting 23:48:05 (3824): No heartbeat from core client for 30 sec - exiting 23:48:06 (3824): No heartbeat from core client for 30 sec - exiting 23:48:07 (3824): No heartbeat from core client for 30 sec - exiting 23:48:08 (3824): No heartbeat from core client for 30 sec - exiting 23:48:10 (3824): No heartbeat from core client for 30 sec - exiting 23:48:11 (3824): No heartbeat from core client for 30 sec - exiting 23:48:12 (3824): No heartbeat from core client for 30 sec - exiting 23:48:13 (3824): No heartbeat from core client for 30 sec - exiting 23:48:14 (3824): No heartbeat from core client for 30 sec - exiting 23:48:15 (3824): No heartbeat from core client for 30 sec - exiting 23:48:16 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:29:10 (1876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:22 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 00:15:28 (2196): No heartbeat from core client for 30 sec - exiting 00:15:29 (2196): No heartbeat from core client for 30 sec - exiting 00:15:30 (2196): No heartbeat from core client for 30 sec - exiting 00:15:31 (2196): No heartbeat from core client for 30 sec - exiting 00:15:32 (2196): No heartbeat from core client for 30 sec - exiting 00:15:33 (2196): No heartbeat from core client for 30 sec - exiting 00:15:34 (2196): No heartbeat from core client for 30 sec - exiting 00:15:35 (2196): No heartbeat from core client for 30 sec - exiting 00:15:36 (2196): No heartbeat from core client for 30 sec - exiting 00:15:37 (2196): No heartbeat from core client for 30 sec - exiting 00:15:39 (2196): No heartbeat from core client for 30 sec - exiting 00:15:40 (2196): No heartbeat from core client for 30 sec - exiting 00:15:41 (2196): No heartbeat from core client for 30 sec - exiting 00:15:42 (2196): No heartbeat from core client for 30 sec - exiting 00:15:43 (2196): No heartbeat from core client for 30 sec - exiting 00:15:44 (2196): No heartbeat from core client for 30 sec - exiting 00:15:45 (2196): No heartbeat from core client for 30 sec - exiting 00:15:46 (2196): No heartbeat from core client for 30 sec - exiting 00:15:47 (2196): No heartbeat from core client for 30 sec - exiting 00:15:48 (2196): No heartbeat from core client for 30 sec - exiting 00:15:50 (2196): No heartbeat from core client for 30 sec - exiting 00:15:51 (2196): No heartbeat from core client for 30 sec - exiting 00:15:52 (2196): No heartbeat from core client for 30 sec - exiting 00:15:53 (2196): No heartbeat from core client for 30 sec - exiting 00:15:54 (2196): No heartbeat from core client for 30 sec - exiting 00:15:55 (2196): No heartbeat from core client for 30 sec - exiting 00:15:56 (2196): No heartbeat from core client for 30 sec - exiting 00:15:57 (2196): No heartbeat from core client for 30 sec - exiting 00:15:58 (2196): No heartbeat from core client for 30 sec - exiting 00:15:59 (2196): No heartbeat from core client for 30 sec - exiting 00:16:00 (2196): No heartbeat from core client for 30 sec - exiting 00:16:02 (2196): No heartbeat from core client for 30 sec - exiting 00:16:03 (2196): No heartbeat from core client for 30 sec - exiting 00:16:04 (2196): No heartbeat from core client for 30 sec - exiting 00:16:05 (2196): No heartbeat from core client for 30 sec - exiting 00:16:06 (2196): No heartbeat from core client for 30 sec - exiting 00:16:07 (2196): No heartbeat from core client for 30 sec - exiting 00:16:08 (2196): No heartbeat from core client for 30 sec - exiting 00:16:09 (2196): No heartbeat from core client for 30 sec - exiting 00:16:10 (2196): No heartbeat from core client for 30 sec - exiting 00:16:11 (2196): No heartbeat from core client for 30 sec - exiting 00:16:13 (2196): No heartbeat from core client for 30 sec - exiting 00:16:14 (2196): No heartbeat from core client for 30 sec - exiting 00:16:15 (2196): No heartbeat from core client for 30 sec - exiting 00:16:16 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:31:10 (3736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=896, iMonCtr=1 Model crash detected, will try to restart... 00:42:23 (896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:48:46 (3104): No heartbeat from core client for 30 sec - exiting 00:48:47 (3104): No heartbeat from core client for 30 sec - exiting 00:48:48 (3104): No heartbeat from core client for 30 sec - exiting 00:48:49 (3104): No heartbeat from core client for 30 sec - exiting 00:48:50 (3104): No heartbeat from core client for 30 sec - exiting 00:48:51 (3104): No heartbeat from core client for 30 sec - exiting 00:48:52 (3104): No heartbeat from core client for 30 sec - exiting 00:48:53 (3104): No heartbeat from core client for 30 sec - exiting 00:48:55 (3104): No heartbeat from core client for 30 sec - exiting 00:48:56 (3104): No heartbeat from core client for 30 sec - exiting 00:48:57 (3104): No heartbeat from core client for 30 sec - exiting 00:48:58 (3104): No heartbeat from core client for 30 sec - exiting 00:48:59 (3104): No heartbeat from core client for 30 sec - exiting 00:49:00 (3104): No heartbeat from core client for 30 sec - exiting 00:49:01 (3104): No heartbeat from core client for 30 sec - exiting 00:49:02 (3104): No heartbeat from core client for 30 sec - exiting 00:49:03 (3104): No heartbeat from core client for 30 sec - exiting 00:49:04 (3104): No heartbeat from core client for 30 sec - exiting 00:49:06 (3104): No heartbeat from core client for 30 sec - exiting 00:49:07 (3104): No heartbeat from core client for 30 sec - exiting 00:49:08 (3104): No heartbeat from core client for 30 sec - exiting 00:49:09 (3104): No heartbeat from core client for 30 sec - exiting 00:49:10 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:01:41 (4408): No heartbeat from core client for 30 sec - exiting 01:01:42 (4408): No heartbeat from core client for 30 sec - exiting 01:01:43 (4408): No heartbeat from core client for 30 sec - exiting 01:01:44 (4408): No heartbeat from core client for 30 sec - exiting 01:01:46 (4408): No heartbeat from core client for 30 sec - exiting 01:01:47 (4408): No heartbeat from core client for 30 sec - exiting 01:01:48 (4408): No heartbeat from core client for 30 sec - exiting 01:01:49 (4408): No heartbeat from core client for 30 sec - exiting 01:01:50 (4408): No heartbeat from core client for 30 sec - exiting 01:01:51 (4408): No heartbeat from core client for 30 sec - exiting 01:01:52 (4408): No heartbeat from core client for 30 sec - exiting 01:01:53 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:54 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 01:12:20 (1412): No heartbeat from core client for 30 sec - exiting 01:12:21 (1412): No heartbeat from core client for 30 sec - exiting 01:12:22 (1412): No heartbeat from core client for 30 sec - exiting 01:12:23 (1412): No heartbeat from core client for 30 sec - exiting 01:12:24 (1412): No heartbeat from core client for 30 sec - exiting 01:12:25 (1412): No heartbeat from core client for 30 sec - exiting 01:12:27 (1412): No heartbeat from core client for 30 sec - exiting 01:12:28 (1412): No heartbeat from core client for 30 sec - exiting 01:12:29 (1412): No heartbeat from core client for 30 sec - exiting 01:12:30 (1412): No heartbeat from core client for 30 sec - exiting 01:12:31 (1412): No heartbeat from core client for 30 sec - exiting 01:12:32 (1412): No heartbeat from core client for 30 sec - exiting 01:12:33 (1412): No heartbeat from core client for 30 sec - exiting 01:12:34 (1412): No heartbeat from core client for 30 sec - exiting 01:12:35 (1412): No heartbeat from core client for 30 sec - exiting 01:12:37 (1412): No heartbeat from core client for 30 sec - exiting 01:12:38 (1412): No heartbeat from core client for 30 sec - exiting 01:12:39 (1412): No heartbeat from core client for 30 sec - exiting 01:12:40 (1412): No heartbeat from core client for 30 sec - exiting 01:12:41 (1412): No heartbeat from core client for 30 sec - exiting 01:12:42 (1412): No heartbeat from core client for 30 sec - exiting 01:12:43 (1412): No heartbeat from core client for 30 sec - exiting 01:12:44 (1412): No heartbeat from core client for 30 sec - exiting 01:12:45 (1412): No heartbeat from core client for 30 sec - exiting 01:12:46 (1412): No heartbeat from core client for 30 sec - exiting 01:12:47 (1412): No heartbeat from core client for 30 sec - exiting 01:12:49 (1412): No heartbeat from core client for 30 sec - exiting 01:12:50 (1412): No heartbeat from core client for 30 sec - exiting 01:12:51 (1412): No heartbeat from core client for 30 sec - exiting 01:12:52 (1412): No heartbeat from core client for 30 sec - exiting 01:12:53 (1412): No heartbeat from core client for 30 sec - exiting 01:12:54 (1412): No heartbeat from core client for 30 sec - exiting 01:12:55 (1412): No heartbeat from core client for 30 sec - exiting 01:12:56 (1412): No heartbeat from core client for 30 sec - exiting 01:12:57 (1412): No heartbeat from core client for 30 sec - exiting 01:12:58 (1412): No heartbeat from core client for 30 sec - exiting 01:13:00 (1412): No heartbeat from core client for 30 sec - exiting 01:13:01 (1412): No heartbeat from core client for 30 sec - exiting 01:13:02 (1412): No heartbeat from core client for 30 sec - exiting 01:13:03 (1412): No heartbeat from core client for 30 sec - exiting 01:13:04 (1412): No heartbeat from core client for 30 sec - exiting 01:13:05 (1412): No heartbeat from core client for 30 sec - exiting 01:13:06 (1412): No heartbeat from core client for 30 sec - exiting 01:13:08 (1412): No heartbeat from core client for 30 sec - exiting 01:13:10 (1412): No heartbeat from core client for 30 sec - exiting 01:13:13 (1412): No heartbeat from core client for 30 sec - exiting 01:13:15 (1412): No heartbeat from core client for 30 sec - exiting 01:13:16 (1412): No heartbeat from core client for 30 sec - exiting 01:13:17 (1412): No heartbeat from core client for 30 sec - exiting 01:13:18 (1412): No heartbeat from core client for 30 sec - exiting 01:13:19 (1412): No heartbeat from core client for 30 sec - exiting 01:13:20 (1412): No heartbeat from core client for 30 sec - exiting 01:13:21 (1412): No heartbeat from core client for 30 sec - exiting 01:13:23 (1412): No heartbeat from core client for 30 sec - exiting 01:13:24 (1412): No heartbeat from core client for 30 sec - exiting 01:13:25 (1412): No heartbeat from core client for 30 sec - exiting 01:13:28 (1412): No heartbeat from core client for 30 sec - exiting 01:13:30 (1412): No heartbeat from core client for 30 sec - exiting 01:13:31 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:53 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:31:14 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:07:48 (2260): No heartbeat from core client for 30 sec - exiting 05:07:50 (2260): No heartbeat from core client for 30 sec - exiting 05:07:51 (2260): No heartbeat from core client for 30 sec - exiting 05:07:52 (2260): No heartbeat from core client for 30 sec - exiting 05:07:53 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:50:33 (5876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:50:34 (5876): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:38:47 (4588): No heartbeat from core client for 30 sec - exiting 16:38:48 (4588): No heartbeat from core client for 30 sec - exiting 16:38:49 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:50:01 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:04 (4600): No heartbeat from core client for 30 sec - exiting 13:09:18 (5328): No heartbeat from core client for 30 sec - exiting 13:09:27 (5328): No heartbeat from core client for 30 sec - exiting 13:14:04 (5328):CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting 13:51:11 (5328): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4308, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jan 2013 01:33:22	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	751,680	1,686,464	2.2436
27 Jan 2013 12:16:41	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	725,760	1,641,002	2.2611
26 Jan 2013 17:21:01	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	699,840	1,594,819	2.2788
24 Jan 2013 18:41:27	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	673,920	1,539,030	2.2837
23 Jan 2013 15:25:34	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	648,000	1,488,490	2.2971
22 Jan 2013 01:04:04	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	622,080	1,440,568	2.3157
21 Jan 2013 10:41:41	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	596,160	1,395,106	2.3402
20 Jan 2013 04:43:56	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	570,240	1,345,100	2.3588
19 Jan 2013 04:24:19	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	544,320	1,277,649	2.3472
18 Jan 2013 11:51:06	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	518,400	1,224,229	2.3616
17 Jan 2013 16:40:34	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	492,480	1,177,712	2.3914
16 Jan 2013 23:17:33	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	466,560	1,122,931	2.4068
15 Jan 2013 18:55:42	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	440,640	1,078,144	2.4468
14 Jan 2013 14:15:38	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	414,720	1,034,231	2.4938
13 Jan 2013 07:09:11	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	388,800	976,302	2.5111
11 Jan 2013 04:23:34	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	362,880	902,022	2.4857
10 Jan 2013 15:10:39	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	336,960	856,402	2.5416
09 Jan 2013 22:31:43	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	311,040	806,894	2.5942
09 Jan 2013 07:45:02	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	285,120	758,707	2.6610
08 Jan 2013 14:05:28	892963	15483836	hadcm3n_za7y_1880_40_008200822_4	259,200	706,311	2.7250