Task 15693837

Name	hadcm3n_3cu6_1940_40_008266458_2
Workunit	8421582
Created	29 Mar 2013, 20:27:54 UTC
Sent	29 Mar 2013, 20:28:17 UTC
Report deadline	29 Jun 2013, 3:55:28 UTC
Received	20 May 2013, 11:57:04 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1072815
Run time	21 days 7 hours 11 min 27 sec
CPU time	20 days 15 hours 52 min 22 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <stderr_txt> 22:45:04 (5544): No heartbeat from core client for 30 sec - exiting 22:45:05 (5544): No heartbeat from core client for 30 sec - exiting 22:45:06 (5544): No heartbeat from core client for 30 sec - exiting 22:45:07 (5544): No heartbeat from core client for 30 sec - exiting 22:45:08 (5544): No heartbeat from core client for 30 sec - exiting 22:45:09 (5544): No heartbeat from core client for 30 sec - exiting 22:45:10 (5544): No heartbeat from core client for 30 sec - exiting 22:45:11 (5544): No heartbeat from core client for 30 sec - exiting 22:45:12 (5544): No heartbeat from core client for 30 sec - exiting 22:45:13 (5544): No heartbeat from core client for 30 sec - exiting 22:45:14 (5544): No heartbeat from core client for 30 sec - exiting 22:45:15 (5544): No heartbeat from core client for 30 sec - exiting 22:45:16 (5544): No heartbeat from core client for 30 sec - exiting 22:45:17 (5544): No heartbeat from core client for 30 sec - exiting 22:45:18 (5544): No heartbeat from core client for 30 sec - exiting 22:45:19 (5544): No heartbeat from core client for 30 sec - exiting 22:45:20 (5544): No heartbeat from core client for 30 sec - exiting 22:45:21 (5544): No heartbeat from core client for 30 sec - exiting 22:45:22 (5544): No heartbeat from core client for 30 sec - exiting 22:45:23 (5544): No heartbeat from core client for 30 sec - exiting 22:45:24 (5544): No heartbeat from core client for 30 sec - exiting 22:45:25 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:45:26 (5544): No heartbeat from core client for 30 sec - exiting 22:45:27 (5544): No heartbeat from core client for 30 sec - exiting 22:45:28 (5544): No heartbeat from core client for 30 sec - exiting 22:45:29 (5544): No heartbeat from core client for 30 sec - exiting 22:45:30 (5544): No heartbeat from core client for 30 sec - exiting 22:45:34 (5544): No heartbeat from core client for 30 sec - exiting 22:45:35 (5544): No heartbeat from core client for 30 sec - exiting 22:45:36 (5544): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 11:18:33 (5196): No heartbeat from core client for 30 sec - exiting 11:18:34 (5196): No heartbeat from core client for 30 sec - exiting 11:18:35 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:36 (5196): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3852, iMonCtr=1 Model crash detected, will try to restart... 12:17:19 (5660): No heartbeat from core client for 30 sec - exiting 12:17:21 (5660): No heartbeat from core client for 30 sec - exiting 12:17:22 (5660): No heartbeat from core client for 30 sec - exiting 12:17:23 (5660): No heartbeat from core client for 30 sec - exiting 12:17:24 (5660): No heartbeat from core client for 30 sec - exiting 12:17:25 (5660): No heartbeat from core client for 30 sec - exiting 12:17:26 (5660): No heartbeat from core client for 30 sec - exiting 12:17:27 (5660): No heartbeat from core client for 30 sec - exiting 12:17:28 (5660): No heartbeat from core client for 30 sec - exiting 12:17:29 (5660): No heartbeat from core client for 30 sec - exiting 12:17:30 (5660): No heartbeat from core client for 30 sec - exiting 12:17:32 (5660): No heartbeat from core client for 30 sec - exiting 12:17:33 (5660): No heartbeat from core client for 30 sec - exiting 12:17:34 (5660): No heartbeat from core client for 30 sec - exiting 12:17:35 (5660): No heartbeat from core client for 30 sec - exiting 12:17:36 (5660): No heartbeat from core client for 30 sec - exiting 12:17:37 (5660): No heartbeat from core client for 30 sec - exiting 12:17:38 (5660): No heartbeat from core client for 30 sec - exiting 12:17:39 (5660): No heartbeat from core client for 30 sec - exiting 12:17:40 (5660): No heartbeat from core client for 30 sec - exiting 12:17:41 (5660): No heartbeat from core client for 30 sec - exiting 12:17:42 (5660): No heartbeat from core client for 30 sec - exiting 12:17:44 (5660): No heartbeat from core client for 30 sec - exiting 12:17:45 (5660): No heartbeat from core client for 30 sec - exiting 12:17:46 (5660): No heartbeat from core client for 30 sec - exiting 12:17:47 (5660): No heartbeat from core client for 30 sec - exiting 12:17:48 (5660): No heartbeat from core client for 30 sec - exiting 12:17:49 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:17:50 (5660): No heartbeat from core client for 30 sec - exiting 12:17:51 (5660): No heartbeat from core client for 30 sec - exiting 12:17:52 (5660): No heartbeat from core client for 30 sec - exiting 12:17:53 (5660): No heartbeat from core client for 30 sec - exiting 12:18:34 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:55:07 (6524): No heartbeat from core client for 30 sec - exiting 18:55:08 (6524): No heartbeat from core client for 30 sec - exiting 18:55:10 (6524): No heartbeat from core client for 30 sec - exiting 18:55:11 (6524): No heartbeat from core client for 30 sec - exiting 18:55:12 (6524): No heartbeat from core client for 30 sec - exiting 18:55:13 (6524): No heartbeat from core client for 30 sec - exiting 18:55:14 (6524): No heartbeat from core client for 30 sec - exiting 18:55:15 (6524): No heartbeat from core client for 30 sec - exiting 18:55:16 (6524): No heartbeat from core client for 30 sec - exiting 18:55:17 (6524): No heartbeat from core client for 30 sec - exiting 18:55:18 (6524): No heartbeat from core client for 30 sec - exiting 18:55:19 (6524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:59 (6216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:38 (7076): No heartbeat from core client for 30 sec - exiting 18:53:39 (7076): No heartbeat from core client for 30 sec - exiting 18:53:40 (7076): No heartbeat from core client for 30 sec - exiting 18:53:41 (7076): No heartbeat from core client for 30 sec - exiting 18:53:42 (7076): No heartbeat from core client for 30 sec - exiting 18:53:43 (7076): No heartbeat from core client for 30 sec - exiting 18:53:44 (7076): No heartbeat from core client for 30 sec - exiting 18:53:45 (7076): No heartbeat from core client for 30 sec - exiting 18:53:46 (7076): No heartbeat from core client for 30 sec - exiting 18:53:48 (7076): No heartbeat from core client for 30 sec - exiting 18:53:49 (7076): No heartbeat from core client for 30 sec - exiting 18:53:50 (7076): No heartbeat from core client for 30 sec - exiting 18:53:51 (7076): No heartbeat from core client for 30 sec - exiting 18:53:52 (7076): No heartbeat from core client for 30 sec - exiting 18:53:53 (7076): No heartbeat from core client for 30 sec - exiting 18:53:54 (7076): No heartbeat from core client for 30 sec - exiting 18:53:55 (7076): No heartbeat from core client for 30 sec - exiting 18:53:56 (7076): No heartbeat from core client for 30 sec - exiting 18:53:57 (7076): No heartbeat from core client for 30 sec - exiting 18:53:58 (7076): No heartbeat from core client for 30 sec - exiting 18:54:00 (7076): No heartbeat from core client for 30 sec - exiting 18:54:01 (7076): No heartbeat from core client for 30 sec - exiting 18:54:02 (7076): No heartbeat from core client for 30 sec - exiting 18:54:03 (7076): No heartbeat from core client for 30 sec - exiting 18:54:04 (7076): No heartbeat from core client for 30 sec - exiting 18:54:05 (7076): No heartbeat from core client for 30 sec - exiting 18:54:06 (7076): No heartbeat from core client for 30 sec - exiting 18:54:07 (7076): No heartbeat from core client for 30 sec - exiting 18:54:08 (7076): No heartbeat from core client for 30 sec - exiting 18:54:09 (7076): No heartbeat from core client for 30 sec - exiting 18:54:11 (7076): No heartbeat from core client for 30 sec - exiting 18:54:12 (7076): No heartbeat from core client for 30 sec - exiting 18:54:13 (7076): No heartbeat from core client for 30 sec - exiting 18:54:14 (7076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:46 (3224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5632, iMonCtr=1 Model crash detected, will try to restart... 18:47:21 (2800): No heartbeat from core client for 30 sec - exiting 18:47:22 (2800): No heartbeat from core client for 30 sec - exiting 18:47:23 (2800): No heartbeat from core client for 30 sec - exiting 18:47:24 (2800): No heartbeat from core client for 30 sec - exiting 18:47:25 (2800): No heartbeat from core client for 30 sec - exiting 18:47:26 (2800): No heartbeat from core client for 30 sec - exiting 18:47:27 (2800): No heartbeat from core client for 30 sec - exiting 18:47:28 (2800): No heartbeat from core client for 30 sec - exiting 18:47:29 (2800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3cu6ko.pje5c10 Error converting file to netcdf: dataout/3cu6ko.pie5c10 Error converting file to netcdf: dataout/3cu6ko.pfe5c10 Error converting file to netcdf: dataout/3cu6ka.phe5c10 Error converting file to netcdf: dataout/3cu6ka.pge5c10 Error converting file to netcdf: dataout/3cu6ka.pee5c10 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:46 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:42:59 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:55:58 (220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7084, iMonCtr=1 Model crash detected, will try to restart... 18:31:02 (2960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6844, iMonCtr=1 Model crash detected, will try to restart... 17:59:28 (6464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6684, iMonCtr=1 Model crash detected, will try to restart... 08:35:33 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:40:06 (3084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:07:35 (5384): No heartbeat from core client for 30 sec - exiting 19:07:37 (5384): No heartbeat from core client for 30 sec - exiting 19:07:38 (5384): No heartbeat from core client for 30 sec - exiting 19:07:39 (5384): No heartbeat from core client for 30 sec - exiting 19:07:40 (5384): No heartbeat from core client for 30 sec - exiting 19:07:41 (5384): No heartbeat from core client for 30 sec - exiting 19:07:42 (5384): No heartbeat from core client for 30 sec - exiting 19:07:43 (5384): No heartbeat from core client for 30 sec - exiting 19:07:44 (5384): No heartbeat from core client for 30 sec - exiting 19:07:45 (5384): No heartbeat from core client for 30 sec - exiting 19:07:46 (5384): No heartbeat from core client for 30 sec - exiting 19:07:47 (5384): No heartbeat from core client for 30 sec - exiting 19:07:48 (5384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:17 (7280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6804, iMonCtr=1 Model crash detected, will try to restart... 20:54:09 (5836): No heartbeat from core client for 30 sec - exiting 20:54:10 (5836): No heartbeat from core client for 30 sec - exiting 20:54:11 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:54:56 (6404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:16:21 (7116): No heartbeat from core client for 30 sec - exiting 00:16:22 (7116): No heartbeat from core client for 30 sec - exiting 00:16:23 (7116): No heartbeat from core client for 30 sec - exiting 00:16:25 (7116): No heartbeat from core client for 30 sec - exiting 00:16:26 (7116): No heartbeat from core client for 30 sec - exiting 00:16:27 (7116): No heartbeat from core client for 30 sec - exiting 00:16:28 (7116): No heartbeat from core client for 30 sec - exiting 00:16:29 (7116): No heartbeat from core client for 30 sec - exiting 00:16:30 (7116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3464, iMonCtr=1 Model crash detected, will try to restart... 19:55:43 (7124): No heartbeat from core client for 30 sec - exiting 19:55:44 (7124): No heartbeat from core client for 30 sec - exiting 19:55:45 (7124): No heartbeat from core client for 30 sec - exiting 19:55:46 (7124): No heartbeat from core client for 30 sec - exiting 19:55:47 (7124): No heartbeat from core client for 30 sec - exiting 19:55:48 (7124): No heartbeat from core client for 30 sec - exiting 19:55:49 (7124): No heartbeat from core client for 30 sec - exiting 19:55:50 (7124): No heartbeat from core client for 30 sec - exiting 19:55:52 (7124): No heartbeat from core client for 30 sec - exiting 19:55:53 (7124): No heartbeat from core client for 30 sec - exiting 19:55:54 (7124): No heartbeat from core client for 30 sec - exiting 19:55:55 (7124): No heartbeat from core client for 30 sec - exiting 19:55:56 (7124): No heartbeat from core client for 30 sec - exiting 19:55:57 (7124): No heartbeat from core client for 30 sec - exiting 19:55:58 (7124): No heartbeat from core client for 30 sec - exiting 19:55:59 (7124): No heartbeat from core client for 30 sec - exiting 19:56:00 (7124): No heartbeat from core client for 30 sec - exiting 19:56:01 (7124): No heartbeat from core client for 30 sec - exiting 19:56:03 (7124): No heartbeat from core client for 30 sec - exiting 19:56:04 (7124): No heartbeat from core client for 30 sec - exiting 19:56:05 (7124): No heartbeat from core client for 30 sec - exiting 19:56:06 (7124): No heartbeat from core client for 30 sec - exiting 19:56:07 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:28:23 (1236): No heartbeat from core client for 30 sec - exiting 19:28:24 (1236): No heartbeat from core client for 30 sec - exiting 19:28:25 (1236): No heartbeat from core client for 30 sec - exiting 19:28:26 (1236): No heartbeat from core client for 30 sec - exiting 19:28:27 (1236): No heartbeat from core client for 30 sec - exiting 19:28:28 (1236): No heartbeat from core client for 30 sec - exiting 19:28:29 (1236): No heartbeat from core client for 30 sec - exiting 19:28:31 (1236): No heartbeat from core client for 30 sec - exiting 19:28:32 (1236): No heartbeat from core client for 30 sec - exiting 19:28:33 (1236): No heartbeat from core client for 30 sec - exiting 19:28:34 (1236): No heartbeat from core client for 30 sec - exiting 19:28:35 (1236): No heartbeat from core client for 30 sec - exiting 19:28:36 (1236): No heartbeat from core client for 30 sec - exiting 19:28:37 (1236): No heartbeat from core client for 30 sec - exiting 19:28:38 (1236): No heartbeat from core client for 30 sec - exiting 19:28:39 (1236): No heartbeat from core client for 30 sec - exiting 19:28:40 (1236): No heartbeat from core client for 30 sec - exiting 19:28:41 (1236): No heartbeat from core client for 30 sec - exiting 19:28:43 (1236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:00:47 (2388): No heartbeat from core client for 30 sec - exiting 19:00:48 (2388): No heartbeat from core client for 30 sec - exiting 19:00:49 (2388): No heartbeat from core client for 30 sec - exiting 19:00:50 (2388): No heartbeat from core client for 30 sec - exiting 19:00:51 (2388): No heartbeat from core client for 30 sec - exiting 19:00:52 (2388): No heartbeat from core client for 30 sec - exiting 19:00:53 (2388): No heartbeat from core client for 30 sec - exiting 19:00:54 (2388): No heartbeat from core client for 30 sec - exiting 19:00:55 (2388): No heartbeat from core client for 30 sec - exiting 19:00:57 (2388): No heartbeat from core client for 30 sec - exiting 19:00:58 (2388): No heartbeat from core client for 30 sec - exiting 19:00:59 (2388): No heartbeat from core client for 30 sec - exiting 19:01:00 (2388): No heartbeat from core client for 30 sec - exiting 19:01:01 (2388): No heartbeat from core client for 30 sec - exiting 19:01:02 (2388): No heartbeat from core client for 30 sec - exiting 19:01:03 (2388): No heartbeat from core client for 30 sec - exiting 19:01:04 (2388): No heartbeat from core client for 30 sec - exiting 19:01:05 (2388): No heartbeat from core client for 30 sec - exiting 19:01:06 (2388): No heartbeat from core client for 30 sec - exiting 19:01:08 (2388): No heartbeat from core client for 30 sec - exiting 19:01:09 (2388): No heartbeat from core client for 30 sec - exiting 19:01:10 (2388): No heartbeat from core client for 30 sec - exiting 19:01:11 (2388): No heartbeat from core client for 30 sec - exiting 19:01:12 (2388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... 22:07:16 (4044): No heartbeat from core client for 30 sec - exiting 22:07:17 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:20:25 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:23:47 (6464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:47:30 (5928): No heartbeat from core client for 30 sec - exiting 20:47:31 (5928): No heartbeat from core client for 30 sec - exiting 20:47:32 (5928): No heartbeat from core client for 30 sec - exiting 20:47:34 (5928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:35 (5928): No heartbeat from core client for 30 sec - exiting 20:48:28 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5752, iMonCtr=1 Model crash detected, will try to restart... 11:38:00 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:36:01 (2180): No heartbeat from core client for 30 sec - exiting 11:36:03 (2180): No heartbeat from core client for 30 sec - exiting 11:36:04 (2180): No heartbeat from core client for 30 sec - exiting 11:36:05 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:06 (2180): No heartbeat from core client for 30 sec - exiting 11:36:07 (2180): No heartbeat from core client for 30 sec - exiting 11:36:08 (2180): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5820, iMonCtr=1 Model crash detected, will try to restart... 11:35:57 (2880): No heartbeat from core client for 30 sec - exiting 11:35:58 (2880): No heartbeat from core client for 30 sec - exiting 11:35:59 (2880): No heartbeat from core client for 30 sec - exiting 11:36:00 (2880): No heartbeat from core client for 30 sec - exiting 11:36:01 (2880): No heartbeat from core client for 30 sec - exiting 11:36:02 (2880): No heartbeat from core client for 30 sec - exiting 11:36:03 (2880): No heartbeat from core client for 30 sec - exiting 11:36:04 (2880): No heartbeat from core client for 30 sec - exiting 11:36:05 (2880): No heartbeat from core client for 30 sec - exiting 11:36:06 (2880): No heartbeat from core client for 30 sec - exiting 11:36:07 (2880): No heartbeat from core client for 30 sec - exiting 11:36:09 (2880): No heartbeat from core client for 30 sec - exiting 11:36:10 (2880): No heartbeat from core client for 30 sec - exiting 11:36:11 (2880): No heartbeat from core client for 30 sec - exiting 11:36:12 (2880): No heartbeat from core client for 30 sec - exiting 11:36:13 (2880): No heartbeat from core client for 30 sec - exiting 11:36:14 (2880): No heartbeat from core client for 30 sec - exiting 11:36:15 (2880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:37:47 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5488, iMonCtr=1 Model crash detected, will try to restart... 10:41:00 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:32:13 (6068): No heartbeat from core client for 30 sec - exiting 11:32:14 (6068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:49:40 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:15 (2996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:07:30 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:22 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6044, iMonCtr=1 Model crash detected, will try to restart... 11:37:43 (1748): No heartbeat from core client for 30 sec - exiting 11:37:44 (1748): No heartbeat from core client for 30 sec - exiting 11:37:45 (1748): No heartbeat from core client for 30 sec - exiting 11:37:46 (1748): No heartbeat from core client for 30 sec - exiting 11:37:47 (1748): No heartbeat from core client for 30 sec - exiting 11:37:48 (1748): No heartbeat from core client for 30 sec - exiting 11:37:49 (1748): No heartbeat from core client for 30 sec - exiting 11:37:51 (1748): No heartbeat from core client for 30 sec - exiting 11:37:52 (1748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7044, iMonCtr=1 Model crash detected, will try to restart... 23:02:21 (7144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:03:12 (6308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:08:05 (3648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:09:11 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:09:19 (4844): No heartbeat from core client for 30 sec - exiting 23:09:20 (4844): No heartbeat from core client for 30 sec - exiting 23:09:21 (4844): No heartbeat from core client for 30 sec - exiting 23:09:22 (4844): No heartbeat from core client for 30 sec - exiting 23:09:23 (4844): No heartbeat from core client for 30 sec - exiting 23:09:24 (4844): No heartbeat from core client for 30 sec - exiting 23:09:25 (4844): No heartbeat from core client for 30 sec - exiting 23:09:26 (4844): No heartbeat from core client for 30 sec - exiting 23:09:27 (4844): No heartbeat from core client for 30 sec - exiting 23:10:27 (3172): No heartbeat from core client for 30 sec - exiting 23:10:28 (3172): No heartbeat from core client for 30 sec - exiting 23:10:29 (3172): No heartbeat from core client for 30 sec - exiting 23:10:30 (3172): No heartbeat from core client for 30 sec - exiting 23:10:31 (3172): No heartbeat from core client for 30 sec - exiting 23:10:32 (3172): No heartbeat from core client for 30 sec - exiting 23:10:33 (3172): No heartbeat from core client for 30 sec - exiting 23:10:34 (3172): No heartbeat from core client for 30 sec - exiting 23:10:35 (3172): No heartbeat from core client for 30 sec - exiting 23:10:36 (3172): No heartbeat from core client for 30 sec - exiting 23:10:37 (3172): No heartbeat from core client for 30 sec - exiting 23:10:39 (3172): No heartbeat from core client for 30 sec - exiting 23:10:40 (3172): No heartbeat from core client for 30 sec - exiting 23:10:41 (3172): No heartbeat from core client for 30 sec - exiting 23:10:42 (3172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:59 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:36:41 (5588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:33:10 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... C19:51:56 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:33 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:34:56 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:36:28 (6408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:53:21 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:22 (5484): No heartbeat from core client for 30 sec - exiting 12:53:23 (5484): No heartbeat from core client for 30 sec - exiting 12:53:24 (5484): No heartbeat from core client for 30 sec - exiting 12:54:26 (4300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 19:18:09 (2884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3cu6ko.pjh5c10 Error converting file to netcdf: dataout/3cu6ko.pih5c10 Error converting file to netcdf: dataout/3cu6ko.pfh5c10 Error converting file to netcdf: dataout/3cu6ka.phh5c10 Error converting file to netcdf: dataout/3cu6ka.pgh5c10 Error converting file to netcdf: dataout/3cu6ka.peh5c10 Error converting file to netcdf: dataout/3cu6ka.pdh5c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:02:06 (4864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:02:53 (4728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:32:24 (1844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:46:34 (3860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4732, iMonCtr=1 Model crash detected, will try to restart... 12:16:41 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:49:49 (1064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:54:29 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 May 2013 12:02:00	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	1,036,800	1,784,547	1.7212
19 May 2013 13:53:55	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	1,010,880	1,740,414	1.7217
18 May 2013 15:46:19	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	984,960	1,696,525	1.7224
17 May 2013 16:48:57	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	959,040	1,652,134	1.7227
14 May 2013 23:01:55	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	933,120	1,607,277	1.7225
13 May 2013 18:33:03	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	907,200	1,562,586	1.7224
12 May 2013 11:57:05	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	881,280	1,518,211	1.7227
11 May 2013 12:39:36	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	855,360	1,474,142	1.7234
10 May 2013 12:42:59	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	829,440	1,429,591	1.7236
09 May 2013 17:02:07	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	803,520	1,384,758	1.7234
08 May 2013 18:54:26	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	777,600	1,339,981	1.7232
06 May 2013 18:44:39	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	751,680	1,295,711	1.7238
04 May 2013 22:27:02	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	725,760	1,251,304	1.7241
04 May 2013 10:41:34	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	699,840	1,206,896	1.7245
03 May 2013 11:34:29	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	673,920	1,162,327	1.7247
02 May 2013 10:56:27	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	648,000	1,117,999	1.7253
01 May 2013 14:50:41	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	622,080	1,073,149	1.7251
30 Apr 2013 15:40:53	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	596,160	1,028,263	1.7248
29 Apr 2013 18:43:16	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	570,240	983,472	1.7247
28 Apr 2013 20:43:09	1072815	15693837	hadcm3n_3cu6_1940_40_008266458_2	544,320	939,168	1.7254