Task 13463809

Name	hadcm3n_ymm1_1900_40_007362163_2
Workunit	7559593
Created	6 Oct 2011, 4:39:54 UTC
Sent	6 Oct 2011, 4:40:17 UTC
Report deadline	5 Jan 2012, 12:07:28 UTC
Received	2 Jan 2012, 15:49:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1070382
Run time	25 days 3 hours 1 min 8 sec
CPU time	18 days 7 hours 3 min 15 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 07:45:05 (6268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4752, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:01:09 (5856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:15:48 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:14:04 (1688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:03:14 (6888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:03:47 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:21 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6488, iMonCtr=1 Model crash detected, will try to restart... 13:29:07 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:55:59 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:13:51 (244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... 21:36:45 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:00:55 (4160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:56 (4160): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:27 (3660): No heartbeat from core client for 30 sec - exiting 21:33:29 (3660): No heartbeat from core client for 30 sec - exiting 21:33:30 (3660): No heartbeat from core client for 30 sec - exiting 21:33:31 (3660): No heartbeat from core client for 30 sec - exiting 21:33:32 (3660): No heartbeat from core client for 30 sec - exiting 21:33:33 (3660): No heartbeat from core client for 30 sec - exiting 21:33:34 (3660): No heartbeat from core client for 30 sec - exiting 21:33:35 (3660): No heartbeat from core client for 30 sec - exiting 21:33:36 (3660): No heartbeat from core client for 30 sec - exiting 21:33:37 (3660): No heartbeat from core client for 30 sec - exiting 21:33:38 (3660): No heartbeat from core client for 30 sec - exiting 21:33:39 (3660): No heartbeat from core client for 30 sec - exiting 21:33:41 (3660): No heartbeat from core client for 30 sec - exiting 21:33:42 (3660): No heartbeat from core client for 30 sec - exiting 21:33:43 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:44 (3660): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:02:25 (5036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:25:45 (6604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:02:07 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:51:38 (6848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:02:43 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:07:37 (2224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:17:45 (5256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:43:40 (3628): No heartbeat from core client for 30 sec - exiting 15:43:41 (3628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:42 (3628): No heartbeat from core client for 30 sec - exiting 19:07:39 (3240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:41 (3240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:15:30 (2396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:31:15 (4404): No heartbeat from core client for 30 sec - exiting 08:31:17 (4404): No heartbeat from core client for 30 sec - exiting 08:31:18 (4404): No heartbeat from core client for 30 sec - exiting 08:31:19 (4404): No heartbeat from core client for 30 sec - exiting 08:31:20 (4404): No heartbeat from core client for 30 sec - exiting 08:31:22 (4404): No heartbeat from core client for 30 sec - exiting 08:31:23 (4404): No heartbeat from core client for 30 sec - exiting 08:31:24 (4404): No heartbeat from core client for 30 sec - exiting 08:31:25 (4404): No heartbeat from core client for 30 sec - exiting 08:31:26 (4404): No heartbeat from core client for 30 sec - exiting 08:31:27 (4404): No heartbeat from core client for 30 sec - exiting 08:31:28 (4404): No heartbeat from core client for 30 sec - exiting 08:31:29 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:30 (4404): No heartbeat from core client for 30 sec - exiting 08:31:31 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:04:38 (4684): No heartbeat from core client for 30 sec - exiting 19:04:39 (4684): No heartbeat from core client for 30 sec - exiting 19:04:40 (4684): No heartbeat from core client for 30 sec - exiting 19:04:41 (4684): No heartbeat from core client for 30 sec - exiting 19:04:42 (4684): No heartbeat from core client for 30 sec - exiting 19:04:43 (4684): No heartbeat from core client for 30 sec - exiting 19:04:44 (4684): No heartbeat from core client for 30 sec - exiting 19:04:45 (4684): No heartbeat from core client for 30 sec - exiting 19:04:46 (4684): No heartbeat from core client for 30 sec - exiting 19:04:47 (4684): No heartbeat from core client for 30 sec - exiting 19:04:48 (4684): No heartbeat from core client for 30 sec - exiting 19:04:49 (4684): No heartbeat from core client for 30 sec - exiting 19:04:50 (4684): No heartbeat from core client for 30 sec - exiting 19:04:51 (4684): No heartbeat from core client for 30 sec - exiting 19:04:52 (4684): No heartbeat from core client for 30 sec - exiting 19:04:53 (4684): No heartbeat from core client for 30 sec - exiting 19:04:54 (4684): No heartbeat from core client for 30 sec - exiting 19:04:55 (4684): No heartbeat from core client for 30 sec - exiting 19:04:56 (4684): No heartbeat from core client for 30 sec - exiting 19:04:57 (4684): No heartbeat from core client for 30 sec - exiting 19:04:58 (4684): No heartbeat from core client for 30 sec - exiting 19:04:59 (4684): No heartbeat from core client for 30 sec - exiting 19:05:00 (4684): No heartbeat from core client for 30 sec - exiting 19:05:01 (4684): No heartbeat from core client for 30 sec - exiting 19:05:02 (4684): No heartbeat from core client for 30 sec - exiting 19:05:03 (4684): No heartbeat from core client for 30 sec - exiting 19:05:04 (4684): No heartbeat from core client for 30 sec - exiting 19:05:05 (4684): No heartbeat from core client for 30 sec - exiting 19:05:06 (4684): No heartbeat from core client for 30 sec - exiting 19:05:07 (4684): No heartbeat from core client for 30 sec - exiting 19:05:08 (4684): No heartbeat from core client for 30 sec - exiting 19:05:09 (4684): No heartbeat from core client for 30 sec - exiting 19:05:10 (4684): No heartbeat from core client for 30 sec - exiting 19:05:11 (4684): No heartbeat from core client for 30 sec - exiting 19:05:12 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:13 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:03:49 (2424): No heartbeat from core client for 30 sec - exiting 21:03:50 (2424): No heartbeat from core client for 30 sec - exiting 21:03:51 (2424): No heartbeat from core client for 30 sec - exiting 21:03:52 (2424): No heartbeat from core client for 30 sec - exiting 21:03:53 (2424): No heartbeat from core client for 30 sec - exiting 21:03:54 (2424): No heartbeat from core client for 30 sec - exiting 21:03:55 (2424): No heartbeat from core client for 30 sec - exiting 21:03:56 (2424): No heartbeat from core client for 30 sec - exiting 21:03:57 (2424): No heartbeat from core client for 30 sec - exiting 21:03:58 (2424): No heartbeat from core client for 30 sec - exiting 21:03:59 (2424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:03 (4840): Can't acquire lockfile (32) - waiting 35s 21:05:26 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:38 (4840): Can't set up shared mem: -1. Will run in standalone mode. 21:05:41 (5024): Can't set up shared mem: -1. Will run in standalone mode. 21:06:00 (1064): Can't acquire lockfile (32) - waiting 35s 21:06:35 (1064): Can't acquire lockfile (32) - exiting 21:06:35 (1064): Error: The process cannot access the file because it is being used by another process. (0x20) 21:07:21 (5940): Can't acquire lockfile (32) - waiting 35s 21:07:56 (5940): Can't acquire lockfile (32) - exiting 21:07:56 (5940): Error: The process cannot access the file because it is being used by another process. (0x20) 21:08:25 (1968): Can't acquire lockfile (32) - waiting 35s 21:09:48 (3688): Can't acquire lockfile (32) - waiting 35s 21:10:23 (3688): Can't acquire lockfile (32) - exiting 21:10:23 (3688): Error: The process cannot access the file because it is being used by another process. (0x20) 07:01:02 (3608): No heartbeat from core client for 30 sec - exiting 07:01:03 (3608): No heartbeat from core client for 30 sec - exiting 07:01:04 (3608): No heartbeat from core client for 30 sec - exiting 07:01:05 (3608): No heartbeat from core client for 30 sec - exiting 07:01:06 (3608): No heartbeat from core client for 30 sec - exiting 07:01:07 (3608): No heartbeat from core client for 30 sec - exiting 07:01:08 (3608): No heartbeat from core client for 30 sec - exiting 07:01:09 (3608): No heartbeat from core client for 30 sec - exiting 07:01:11 (3608): No heartbeat from core client for 30 sec - exiting 07:01:12 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:09:33 (2056): No heartbeat from core client for 30 sec - exiting 08:09:34 (2056): No heartbeat from core client for 30 sec - exiting 08:09:35 (2056): No heartbeat from core client for 30 sec - exiting 08:09:36 (2056): No heartbeat from core client for 30 sec - exiting 08:09:37 (2056): No heartbeat from core client for 30 sec - exiting 08:09:38 (2056): No heartbeat from core client for 30 sec - exiting 08:09:40 (2056): No heartbeat from core client for 30 sec - exiting 08:09:41 (2056): No heartbeat from core client for 30 sec - exiting 08:09:42 (2056): No heartbeat from core client for 30 sec - exiting 08:09:43 (2056): No heartbeat from core client for 30 sec - exiting 08:09:44 (2056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:45 (2056): No heartbeat from core client for 30 sec - exiting 08:09:46 (2056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:06:26 (4852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:22:56 (6220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:18:01 (3188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:45:24 (5856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:54:20 (6976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:56:06 (2680): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:48:54 (2412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:19:55 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:43:40 (3544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:44:28 (5496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:29:57 (3648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:31:09 (4264): No heartbeat from core client for 30 sec - exiting 08:31:11 (4264): No heartbeat from core client for 30 sec - exiting 08:31:12 (4264): No heartbeat from core client for 30 sec - exiting 08:31:13 (4264): No heartbeat from core client for 30 sec - exiting 08:31:14 (4264): No heartbeat from core client for 30 sec - exiting 08:31:15 (4264): No heartbeat from core client for 30 sec - exiting 08:31:16 (4264): No heartbeat from core client for 30 sec - exiting 08:31:17 (4264): No heartbeat from core client for 30 sec - exiting 08:31:18 (4264): No heartbeat from core client for 30 sec - exiting 08:31:20 (4264): No heartbeat from core client for 30 sec - exiting 08:31:21 (4264): No heartbeat from core client for 30 sec - exiting 08:31:22 (4264): No heartbeat from core client for 30 sec - exiting 08:31:23 (4264): No heartbeat from core client for 30 sec - exiting 08:31:24 (4264): No heartbeat from core client for 30 sec - exiting 08:31:25 (4264): No heartbeat from core client for 30 sec - exiting 08:31:26 (4264): No heartbeat from core client for 30 sec - exiting 08:31:27 (4264): No heartbeat from core client for 30 sec - exiting 08:31:28 (4264): No heartbeat from core client for 30 sec - exiting 08:31:29 (4264): No heartbeat from core client for 30 sec - exiting 08:31:31 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:41:17 (2416): No heartbeat from core client for 30 sec - exiting 12:41:19 (2416): No heartbeat from core client for 30 sec - exiting 12:41:20 (2416): No heartbeat from core client for 30 sec - exiting 12:41:21 (2416): No heartbeat from core client for 30 sec - exiting 12:41:22 (2416): No heartbeat from core client for 30 sec - exiting 12:41:23 (2416): No heartbeat from core client for 30 sec - exiting 12:41:24 (2416): No heartbeat from core client for 30 sec - exiting 12:41:25 (2416): No heartbeat from core client for 30 sec - exiting 12:41:26 (2416): No heartbeat from core client for 30 sec - exiting 12:41:28 (2416): No heartbeat from core client for 30 sec - exiting 12:41:29 (2416): No heartbeat from core client for 30 sec - exiting 12:41:30 (2416): No heartbeat from core client for 30 sec - exiting 12:41:31 (2416): No heartbeat from core client for 30 sec - exiting 12:41:32 (2416): No heartbeat from core client for 30 sec - exiting 12:41:33 (2416): No heartbeat from core client for 30 sec - exiting 12:41:34 (2416): No heartbeat from core client for 30 sec - exiting 12:41:35 (2416): No heartbeat from core client for 30 sec - exiting 12:41:36 (2416): No heartbeat from core client for 30 sec - exiting 12:41:37 (2416): No heartbeat from core client for 30 sec - exiting 12:41:39 (2416): No heartbeat from core client for 30 sec - exiting 12:41:40 (2416): No heartbeat from core client for 30 sec - exiting 12:41:41 (2416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:14:30 (1404): No heartbeat from core client for 30 sec - exiting 22:14:32 (1404): No heartbeat from core client for 30 sec - exiting 22:14:33 (1404): No heartbeat from core client for 30 sec - exiting 22:14:34 (1404): No heartbeat from core client for 30 sec - exiting 22:14:35 (1404): No heartbeat from core client for 30 sec - exiting 22:14:36 (1404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:14:37 (1404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 17:56:45 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:56:58 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:08:19 (5388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:32:18 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:00:12 (6764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:51:00 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:17:09 (1964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:22:22 (1232): No heartbeat from core client for 30 sec - exiting 09:22:23 (1232): No heartbeat from core client for 30 sec - exiting 09:22:24 (1232): No heartbeat from core client for 30 sec - exiting 09:22:26 (1232): No heartbeat from core client for 30 sec - exiting 09:22:27 (1232): No heartbeat from core client for 30 sec - exiting 09:22:28 (1232): No heartbeat from core client for 30 sec - exiting 09:22:29 (1232): No heartbeat from core client for 30 sec - exiting 09:22:30 (1232): No heartbeat from core client for 30 sec - exiting 09:22:31 (1232): No heartbeat from core client for 30 sec - exiting 09:22:32 (1232): No heartbeat from core client for 30 sec - exiting 09:22:34 (1232): No heartbeat from core client for 30 sec - exiting 09:22:35 (1232): No heartbeat from core client for 30 sec - exiting 09:22:36 (1232): No heartbeat from core client for 30 sec - exiting 09:22:37 (1232): No heartbeat from core client for 30 sec - exiting 09:22:38 (1232): No heartbeat from core client for 30 sec - exiting 09:22:40 (1232): No heartbeat from core client for 30 sec - exiting 09:22:41 (1232): No heartbeat from core client for 30 sec - exiting 09:22:42 (1232): No heartbeat from core client for 30 sec - exiting 09:22:43 (1232): No heartbeat from core client for 30 sec - exiting 09:22:44 (1232): No heartbeat from core client for 30 sec - exiting 09:22:45 (1232): No heartbeat from core client for 30 sec - exiting 09:22:46 (1232): No heartbeat from core client for 30 sec - exiting 09:22:47 (1232): No heartbeat from core client for 30 sec - exiting 09:22:48 (1232): No heartbeat from core client for 30 sec - exiting 09:22:50 (1232): No heartbeat from core client for 30 sec - exiting 09:22:51 (1232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:52 (1232): No heartbeat from core client for 30 sec - exiting 09:24:16 (4388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:34:42 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:43 (4848): No heartbeat from core client for 30 sec - exiting 07:34:44 (4848): No heartbeat from core client for 30 sec - exiting 07:34:45 (4848): No heartbeat from core client for 30 sec - exiting 07:34:46 (4848): No heartbeat from core client for 30 sec - exiting 07:34:47 (4848): No heartbeat from core client for 30 sec - exiting 07:34:49 (4848): No heartbeat from core client for 30 sec - exiting 07:34:50 (4848): No heartbeat from core client for 30 sec - exiting 07:34:51 (4848): No heartbeat from core client for 30 sec - exiting 07:34:52 (4848): No heartbeat from core client for 30 sec - exiting 07:34:53 (4848): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 15:23:46 (3104): No heartbeat from core client for 30 sec - exiting 15:23:47 (3104): No heartbeat from core client for 30 sec - exiting 15:23:48 (3104): No heartbeat from core client for 30 sec - exiting 15:23:49 (3104): No heartbeat from core client for 30 sec - exiting 15:23:50 (3104): No heartbeat from core client for 30 sec - exiting 15:23:51 (3104): No heartbeat from core client for 30 sec - exiting 15:23:52 (3104): No heartbeat from core client for 30 sec - exiting 15:23:53 (3104): No heartbeat from core client for 30 sec - exiting 15:23:54 (3104): No heartbeat from core client for 30 sec - exiting 15:23:55 (3104): No heartbeat from core client for 30 sec - exiting 15:23:56 (3104): No heartbeat from core client for 30 sec - exiting 15:23:58 (3104): No heartbeat from core client for 30 sec - exiting 15:23:59 (3104): No heartbeat from core client for 30 sec - exiting 15:24:00 (3104): No heartbeat from core client for 30 sec - exiting 15:24:01 (3104): No heartbeat from core client for 30 sec - exiting 15:24:02 (3104): No heartbeat from core client for 30 sec - exiting 15:24:03 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:24:04 (3104): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:32:17 (2816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:05:04 (2000): No heartbeat from core client for 30 sec - exiting 09:05:05 (2000): No heartbeat from core client for 30 sec - exiting 09:05:06 (2000): No heartbeat from core client for 30 sec - exiting 09:05:07 (2000): No heartbeat from core client for 30 sec - exiting 09:05:09 (2000): No heartbeat from core client for 30 sec - exiting 09:05:10 (2000): No heartbeat from core client for 30 sec - exiting 09:05:11 (2000): No heartbeat from core client for 30 sec - exiting 09:05:12 (2000): No heartbeat from core client for 30 sec - exiting 09:05:13 (2000): No heartbeat from core client for 30 sec - exiting 09:05:14 (2000): No heartbeat from core client for 30 sec - exiting 09:05:15 (2000): No heartbeat from core client for 30 sec - exiting 09:05:16 (2000): No heartbeat from core client for 30 sec - exiting 09:05:17 (2000): No heartbeat from core client for 30 sec - exiting 09:05:18 (2000): No heartbeat from core client for 30 sec - exiting 09:05:19 (2000): No heartbeat from core client for 30 sec - exiting 09:05:21 (2000): No heartbeat from core client for 30 sec - exiting 09:05:22 (2000): No heartbeat from core client for 30 sec - exiting 09:05:23 (2000): No heartbeat from core client for 30 sec - exiting 09:05:24 (2000): No heartbeat from core client for 30 sec - exiting 09:05:25 (2000): No heartbeat from core client for 30 sec - exiting 09:05:26 (2000): No heartbeat from core client for 30 sec - exiting 09:05:28 (2000): No heartbeat from core client for 30 sec - exiting 09:05:29 (2000): No heartbeat from core client for 30 sec - exiting 09:05:30 (2000): No heartbeat from core client for 30 sec - exiting 09:05:31 (2000): No heartbeat from core client for 30 sec - exiting 09:05:33 (2000): No heartbeat from core client for 30 sec - exiting 09:05:34 (2000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:35 (2000): No heartbeat from core client for 30 sec - exiting 09:05:36 (2000): No heartbeat from core client for 30 sec - exiting 09:05:37 (2000): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jan 2012 15:51:27	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	777,600	1,580,592	2.0327
22 Dec 2011 23:03:17	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	751,680	1,530,003	2.0354
19 Dec 2011 07:07:16	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	725,760	1,473,797	2.0307
14 Dec 2011 07:57:30	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	699,840	1,427,652	2.0400
12 Dec 2011 00:00:58	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	673,920	1,381,248	2.0496
07 Dec 2011 23:04:48	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	648,000	1,333,807	2.0583
05 Dec 2011 03:24:41	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	622,080	1,263,429	2.0310
30 Nov 2011 23:02:12	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	596,160	1,209,702	2.0292
26 Nov 2011 04:48:38	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	570,240	1,154,371	2.0244
21 Nov 2011 23:01:08	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	544,320	1,102,712	2.0259
18 Nov 2011 05:03:11	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	518,400	1,043,284	2.0125
16 Nov 2011 04:22:57	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	492,480	995,183	2.0208
15 Nov 2011 18:25:18	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	466,560	944,794	2.0250
15 Nov 2011 18:25:18	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	440,640	885,871	2.0104
15 Nov 2011 18:25:18	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	414,720	825,824	1.9913
08 Nov 2011 04:26:13	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	388,800	762,798	1.9619
06 Nov 2011 05:36:04	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	362,880	720,876	1.9865
04 Nov 2011 22:19:45	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	336,960	668,962	1.9853
02 Nov 2011 07:07:07	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	311,040	625,575	2.0112
01 Nov 2011 02:37:33	1070382	13463809	hadcm3n_ymm1_1900_40_007362163_2	285,120	575,800	2.0195