Task 15724761

Name	hadcm3n_o78q_2140_40_008269507_2
Workunit	8424631
Created	15 Apr 2013, 1:52:30 UTC
Sent	15 Apr 2013, 1:52:34 UTC
Report deadline	15 Jul 2013, 9:19:45 UTC
Received	31 May 2013, 4:20:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1106245
Run time	17 days 15 hours 57 min 15 sec
CPU time	16 days 11 hours 41 min 13 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6744, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3964, iMonCtr=1 Model crash detected, will try to restart... 09:17:40 (7080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:18 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:51 (8948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:06:10 (8068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:02 (7664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:03 (7664): No heartbeat from core client for 30 sec - exiting 07:11:04 (7664): No heartbeat from core client for 30 sec - exiting 07:11:05 (7664): No heartbeat from core client for 30 sec - exiting 07:11:06 (7664): No heartbeat from core client for 30 sec - exiting 07:11:07 (7664): No heartbeat from core client for 30 sec - exiting 07:11:08 (7664): No heartbeat from core client for 30 sec - exiting 07:11:09 (7664): No heartbeat from core client for 30 sec - exiting 07:11:10 (7664): No heartbeat from core client for 30 sec - exiting 07:11:11 (7664): No heartbeat from core client for 30 sec - exiting 07:11:12 (7664): No heartbeat from core client for 30 sec - exiting 07:12:28 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:10 (8284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:19:33 (4520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:40:09 (7756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:40:55 (7352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:05 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:26:34 (8664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:52:34 (844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:52:35 (844): No heartbeat from core client for 30 sec - exiting 04:52:36 (844): No heartbeat from core client for 30 sec - exiting 04:52:37 (844): No heartbeat from core client for 30 sec - exiting 04:52:38 (844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:45:44 (9400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:52:14 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:52:15 (3104): No heartbeat from core client for 30 sec - exiting 13:52:16 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:26:48 (7568): No heartbeat from core client for 30 sec - exiting 16:26:49 (7568): No heartbeat from core client for 30 sec - exiting 16:26:51 (7568): No heartbeat from core client for 30 sec - exiting 16:26:53 (7568): No heartbeat from core client for 30 sec - exiting 16:26:54 (7568): No heartbeat from core client for 30 sec - exiting 16:26:56 (7568): No heartbeat from core client for 30 sec - exiting 16:26:57 (7568): No heartbeat from core client for 30 sec - exiting 16:26:58 (7568): No heartbeat from core client for 30 sec - exiting 16:27:00 (7568): No heartbeat from core client for 30 sec - exiting 16:27:01 (7568): No heartbeat from core client for 30 sec - exiting 16:27:03 (7568): No heartbeat from core client for 30 sec - exiting 16:27:04 (7568): No heartbeat from core client for 30 sec - exiting 16:27:06 (7568): No heartbeat from core client for 30 sec - exiting 16:27:07 (7568): No heartbeat from core client for 30 sec - exiting 16:27:08 (7568): No heartbeat from core client for 30 sec - exiting 16:27:10 (7568): No heartbeat from core client for 30 sec - exiting 16:27:12 (7568): No heartbeat from core client for 30 sec - exiting 16:27:13 (7568): No heartbeat from core client for 30 sec - exiting 16:27:15 (7568): No heartbeat from core client for 30 sec - exiting 16:27:16 (7568): No heartbeat from core client for 30 sec - exiting 16:27:18 (7568): No heartbeat from core client for 30 sec - exiting 16:27:19 (7568): No heartbeat from core client for 30 sec - exiting 16:27:20 (7568): No heartbeat from core client for 30 sec - exiting 16:27:22 (7568): No heartbeat from core client for 30 sec - exiting 16:27:23 (7568): No heartbeat from core client for 30 sec - exiting 16:27:25 (7568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:12:53 (8424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:54 (8424): No heartbeat from core client for 30 sec - exiting 09:16:42 (8556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:16:43 (8556): No heartbeat from core client for 30 sec - exiting 09:16:44 (8556): No heartbeat from core client for 30 sec - exiting 09:16:45 (8556): No heartbeat from core client for 30 sec - exiting 09:18:35 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:30:10 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:48:36 (6308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:37 (6308): No heartbeat from core client for 30 sec - exiting 14:48:38 (6308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:05:30 (2880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:23:09 (8512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:24:03 (8552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:52:22 (10596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:40 (9424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:37:45 (7432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:51:49 (1668): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:21:15 (9108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 06:22:59 (3356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:48 (2824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:19:08 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 05:14:00 PM No files match the supplied pattern. MainError: 05:14:00 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 01:44:28 PM No files match the supplied pattern. MainError: 01:44:28 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:04:59 (7520): No heartbeat from core client for 30 sec - exiting 09:05:00 (7520): No heartbeat from core client for 30 sec - exiting 09:05:01 (7520): No heartbeat from core client for 30 sec - exiting 09:05:02 (7520): No heartbeat from core client for 30 sec - exiting 09:05:04 (7520): No heartbeat from core client for 30 sec - exiting 09:05:05 (7520): No heartbeat from core client for 30 sec - exiting 09:05:06 (7520): No heartbeat from core client for 30 sec - exiting 09:05:07 (7520): No heartbeat from core client for 30 sec - exiting 09:05:08 (7520): No heartbeat from core client for 30 sec - exiting 09:05:10 (7520): No heartbeat from core client for 30 sec - exiting 09:05:11 (7520): No heartbeat from core client for 30 sec - exiting 09:05:12 (7520): No heartbeat from core client for 30 sec - exiting 09:05:13 (7520): No heartbeat from core client for 30 sec - exiting 09:05:14 (7520): No heartbeat from core client for 30 sec - exiting 09:05:15 (7520): No heartbeat from core client for 30 sec - exiting 09:05:16 (7520): No heartbeat from core client for 30 sec - exiting 09:05:17 (7520): No heartbeat from core client for 30 sec - exiting 09:05:18 (7520): No heartbeat from core client for 30 sec - exiting 09:05:19 (7520): No heartbeat from core client for 30 sec - exiting 09:05:20 (7520): No heartbeat from core client for 30 sec - exiting 09:05:22 (7520): No heartbeat from core client for 30 sec - exiting 09:05:23 (7520): No heartbeat from core client for 30 sec - exiting 09:05:24 (7520): No heartbeat from core client for 30 sec - exiting 09:05:25 (7520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:05 (5020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:11:39 AM No files match the supplied pattern. MainError: 09:11:39 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:11:29 (7068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:27 (9884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:28 (9884): No heartbeat from core client for 30 sec - exiting 13:01:29 (9884): No heartbeat from core client for 30 sec - exiting 13:01:30 (9884): No heartbeat from core client for 30 sec - exiting 13:01:31 (9884): No heartbeat from core client for 30 sec - exiting 13:01:33 (9884): No heartbeat from core client for 30 sec - exiting 13:01:34 (9884): No heartbeat from core client for 30 sec - exiting 13:01:35 (9884): No heartbeat from core client for 30 sec - exiting 13:01:36 (9884): No heartbeat from core client for 30 sec - exiting 13:01:37 (9884): No heartbeat from core client for 30 sec - exiting 13:01:38 (9884): No heartbeat from core client for 30 sec - exiting 13:07:01 (8616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:07:02 (8616): No heartbeat from core client for 30 sec - exiting 13:07:03 (8616): No heartbeat from core client for 30 sec - exiting 13:07:05 (8616): No heartbeat from core client for 30 sec - exiting 13:07:06 (8616): No heartbeat from core client for 30 sec - exiting 13:07:07 (8616): No heartbeat from core client for 30 sec - exiting 13:07:08 (8616): No heartbeat from core client for 30 sec - exiting 13:07:09 (8616): No heartbeat from core client for 30 sec - exiting 13:07:10 (8616): No heartbeat from core client for 30 sec - exiting 13:07:11 (8616): No heartbeat from core client for 30 sec - exiting 13:07:12 (8616): No heartbeat from core client for 30 sec - exiting 13:14:48 (9744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:41 (8864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:43 (8864): No heartbeat from core client for 30 sec - exiting 13:21:44 (8864): No heartbeat from core client for 30 sec - exiting 13:21:45 (8864): No heartbeat from core client for 30 sec - exiting 13:21:46 (8864): No heartbeat from core client for 30 sec - exiting 13:21:47 (8864): No heartbeat from core client for 30 sec - exiting 13:21:48 (8864): No heartbeat from core client for 30 sec - exiting 13:21:49 (8864): No heartbeat from core client for 30 sec - exiting 13:21:50 (8864): No heartbeat from core client for 30 sec - exiting 13:21:51 (8864): No heartbeat from core client for 30 sec - exiting 13:21:52 (8864): No heartbeat from core client for 30 sec - exiting 13:24:58 (8504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:59 (8504): No heartbeat from core client for 30 sec - exiting 13:25:00 (8504): No heartbeat from core client for 30 sec - exiting 13:25:01 (8504): No heartbeat from core client for 30 sec - exiting 13:25:02 (8504): No heartbeat from core client for 30 sec - exiting 13:25:03 (8504): No heartbeat from core client for 30 sec - exiting 13:25:05 (8504): No heartbeat from core client for 30 sec - exiting 13:25:06 (8504): No heartbeat from core client for 30 sec - exiting 13:25:07 (8504): No heartbeat from core client for 30 sec - exiting 13:25:08 (8504): No heartbeat from core client for 30 sec - exiting 13:25:09 (8504): No heartbeat from core client for 30 sec - exiting 13:29:17 (6556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:18 (6556): No heartbeat from core client for 30 sec - exiting 13:33:11 (5980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:33:12 (5980): No heartbeat from core client for 30 sec - exiting 13:33:13 (5980): No heartbeat from core client for 30 sec - exiting 13:33:14 (5980): No heartbeat from core client for 30 sec - exiting 13:33:15 (5980): No heartbeat from core client for 30 sec - exiting 13:33:16 (5980): No heartbeat from core client for 30 sec - exiting 13:33:18 (5980): No heartbeat from core client for 30 sec - exiting 13:33:19 (5980): No heartbeat from core client for 30 sec - exiting 13:33:20 (5980): No heartbeat from core client for 30 sec - exiting 13:33:21 (5980): No heartbeat from core client for 30 sec - exiting 13:33:22 (5980): No heartbeat from core client for 30 sec - exiting 13:35:04 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:35:05 (6140): No heartbeat from core client for 30 sec - exiting 13:35:06 (6140): No heartbeat from core client for 30 sec - exiting 13:35:07 (6140): No heartbeat from core client for 30 sec - exiting 13:35:09 (6140): No heartbeat from core client for 30 sec - exiting 13:35:10 (6140): No heartbeat from core client for 30 sec - exiting 13:35:11 (6140): No heartbeat from core client for 30 sec - exiting 13:35:12 (6140): No heartbeat from core client for 30 sec - exiting 13:35:13 (6140): No heartbeat from core client for 30 sec - exiting 13:35:14 (6140): No heartbeat from core client for 30 sec - exiting 13:35:15 (6140): No heartbeat from core client for 30 sec - exiting 13:39:34 (160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:36 (160): No heartbeat from core client for 30 sec - exiting 13:39:37 (160): No heartbeat from core client for 30 sec - exiting 13:39:38 (160): No heartbeat from core client for 30 sec - exiting 13:39:39 (160): No heartbeat from core client for 30 sec - exiting 13:39:40 (160): No heartbeat from core client for 30 sec - exiting 13:39:41 (160): No heartbeat from core client for 30 sec - exiting 13:42:17 (8784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:47:21 (2164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:47:22 (2164): No heartbeat from core client for 30 sec - exiting 13:47:24 (2164): No heartbeat from core client for 30 sec - exiting 13:47:25 (2164): No heartbeat from core client for 30 sec - exiting 13:47:26 (2164): No heartbeat from core client for 30 sec - exiting 13:47:27 (2164): No heartbeat from core client for 30 sec - exiting 13:47:28 (2164): No heartbeat from core client for 30 sec - exiting 13:47:29 (2164): No heartbeat from core client for 30 sec - exiting 13:47:30 (2164): No heartbeat from core client for 30 sec - exiting 13:47:31 (2164): No heartbeat from core client for 30 sec - exiting 13:47:32 (2164): No heartbeat from core client for 30 sec - exiting 13:49:15 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:49:16 (7648): No heartbeat from core client for 30 sec - exiting 13:49:18 (7648): No heartbeat from core client for 30 sec - exiting 13:49:19 (7648): No heartbeat from core client for 30 sec - exiting 13:49:20 (7648): No heartbeat from core client for 30 sec - exiting 13:49:21 (7648): No heartbeat from core client for 30 sec - exiting 13:49:22 (7648): No heartbeat from core client for 30 sec - exiting 13:49:23 (7648): No heartbeat from core client for 30 sec - exiting 13:49:24 (7648): No heartbeat from core client for 30 sec - exiting 13:49:25 (7648): No heartbeat from core client for 30 sec - exiting 13:49:26 (7648): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 13:51:12 (8924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:53:14 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:53:15 (6568): No heartbeat from core client for 30 sec - exiting 13:53:16 (6568): No heartbeat from core client for 30 sec - exiting 13:53:17 (6568): No heartbeat from core client for 30 sec - exiting 13:53:18 (6568): No heartbeat from core client for 30 sec - exiting 13:53:20 (6568): No heartbeat from core client for 30 sec - exiting 13:53:21 (6568): No heartbeat from core client for 30 sec - exiting 13:53:22 (6568): No heartbeat from core client for 30 sec - exiting 13:53:23 (6568): No heartbeat from core client for 30 sec - exiting 13:53:24 (6568): No heartbeat from core client for 30 sec - exiting 13:53:25 (6568): No heartbeat from core client for 30 sec - exiting 13:55:07 (652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 11:26:27 (10104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:43 (9568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:30:36 (9304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:30:37 (9304): No heartbeat from core client for 30 sec - exiting 11:30:38 (9304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... MainError: 08:52:53 AM No files match the supplied pattern. MainError: 08:52:53 AM No files match the supplied pattern. 19:21:04 (9432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:50:37 (10652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 04:42:23 PM No files match the supplied pattern. MainError: 04:42:23 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 12:02:18 AM No files match the supplied pattern. MainError: 12:02:18 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 03:29:52 AM No files match the supplied pattern. MainError: 03:29:52 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:48:34 PM No files match the supplied pattern. MainError: 09:48:34 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 04:50:06 PM No files match the supplied pattern. MainError: 04:50:06 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 07:26:08 AM No files match the supplied pattern. MainError: 07:26:08 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/o78qka.ph11c10 Error converting file to netcdf: dataout/o78qka.pg11c10 Error converting file to netcdf: dataout/o78qka.pe11c10 MainError: 05:45:23 PM No files match the supplied pattern. MainError: 05:45:23 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 May 2013 17:49:49	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	777,600	1,422,172	1.8289
29 May 2013 07:27:32	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	751,680	1,385,257	1.8429
28 May 2013 16:52:08	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	725,760	1,341,215	1.8480
27 May 2013 21:48:56	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	699,840	1,292,295	1.8466
27 May 2013 03:32:05	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	673,920	1,246,829	1.8501
26 May 2013 00:02:38	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	648,000	1,199,775	1.8515
23 May 2013 16:46:52	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	622,080	1,152,788	1.8531
22 May 2013 08:53:25	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	596,160	1,106,751	1.8565
20 May 2013 09:16:05	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	570,240	1,061,458	1.8614
17 May 2013 13:47:18	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	544,320	1,015,510	1.8656
16 May 2013 17:17:14	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	518,400	965,583	1.8626
15 May 2013 04:44:07	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	492,480	917,254	1.8625
13 May 2013 22:34:26	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	466,560	871,103	1.8671
13 May 2013 06:22:21	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	440,640	824,047	1.8701
12 May 2013 03:44:20	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	414,720	776,148	1.8715
11 May 2013 14:05:07	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	388,800	729,359	1.8759
10 May 2013 22:32:20	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	362,880	681,297	1.8775
09 May 2013 06:30:33	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	336,960	632,956	1.8784
08 May 2013 16:41:05	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	311,040	584,395	1.8788
08 May 2013 04:02:52	1106245	15724761	hadcm3n_o78q_2140_40_008269507_2	285,120	538,928	1.8902