Task 15854650

Name	hadcm3n_3ee2_2020_40_008393116_0
Workunit	8543975
Created	21 Jun 2013, 21:09:11 UTC
Sent	24 Jun 2013, 2:16:27 UTC
Report deadline	23 Sep 2013, 9:43:38 UTC
Received	17 Aug 2013, 10:29:28 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1103902
Run time	54 days 0 hours 48 min 42 sec
CPU time	48 days 9 hours 59 min 10 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	1.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> 08:19:45 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:46 (3124): No heartbeat from core client for 30 sec - exiting 08:19:47 (3124): No heartbeat from core client for 30 sec - exiting Ocean Restart file copy failed on 3ee2ko.dam1bd0 Suspended CPDN Monitor - Suspend request from BOINC... 08:10:40 (6032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3ee2ko.pjm3c10 Error converting file to netcdf: dataout/3ee2ko.pim3c10 Error converting file to netcdf: dataout/3ee2ko.pfm3c10 Error converting file to netcdf: dataout/3ee2ka.phm3c10 Error converting file to netcdf: dataout/3ee2ka.pgm3c10 Error converting file to netcdf: dataout/3ee2ka.pem3c10 Error converting file to netcdf: dataout/3ee2ka.pdm3c10 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ee2ko.dam6790 Ocean Restart file copy failed on 3ee2ko.dam75r0 04:34:52 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:34:53 (4344): No heartbeat from core client for 30 sec - exiting 04:34:54 (4344): No heartbeat from core client for 30 sec - exiting 04:34:55 (4344): No heartbeat from core client for 30 sec - exiting 04:34:56 (4344): No heartbeat from core client for 30 sec - exiting 16:36:21 (3380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ee2ko.dam93l0 Atmos Hold Restart file rename failed on atmos_restart.hold 09:30:20 (588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:21 (588): No heartbeat from core client for 30 sec - exiting 09:30:22 (588): No heartbeat from core client for 30 sec - exiting 09:30:23 (588): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 07:44:44 (3612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:45 (3612): No heartbeat from core client for 30 sec - exiting 07:44:46 (3612): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 05:32:52 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:54 (4964): No heartbeat from core client for 30 sec - exiting 05:32:55 (4964): No heartbeat from core client for 30 sec - exiting 05:32:56 (4964): No heartbeat from core client for 30 sec - exiting 06:43:16 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:43:18 (4796): No heartbeat from core client for 30 sec - exiting 06:43:19 (4796): No heartbeat from core client for 30 sec - exiting 06:43:20 (4796): No heartbeat from core client for 30 sec - exiting 06:47:27 (2824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:31 (2056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:29 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:30 (5436): No heartbeat from core client for 30 sec - exiting 07:12:31 (5436): No heartbeat from core client for 30 sec - exiting 07:12:32 (5436): No heartbeat from core client for 30 sec - exiting 07:17:42 (2604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:44 (2604): No heartbeat from core client for 30 sec - exiting 07:17:45 (2604): No heartbeat from core client for 30 sec - exiting 07:17:46 (2604): No heartbeat from core client for 30 sec - exiting 07:17:47 (2604): No heartbeat from core client for 30 sec - exiting 18:13:06 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:08 (5408): No heartbeat from core client for 30 sec - exiting 18:13:09 (5408): No heartbeat from core client for 30 sec - exiting 18:13:10 (5408): No heartbeat from core client for 30 sec - exiting 01:19:07 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:19:09 (5604): No heartbeat from core client for 30 sec - exiting 01:19:10 (5604): No heartbeat from core client for 30 sec - exiting 08:25:02 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:25:03 (4964): No heartbeat from core client for 30 sec - exiting 08:25:04 (4964): No heartbeat from core client for 30 sec - exiting 08:25:05 (4964): No heartbeat from core client for 30 sec - exiting 19:46:24 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:25 (4716): No heartbeat from core client for 30 sec - exiting 19:46:26 (4716): No heartbeat from core client for 30 sec - exiting 19:46:27 (4716): No heartbeat from core client for 30 sec - exiting 20:22:04 (5876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:22:06 (5876): No heartbeat from core client for 30 sec - exiting 20:22:07 (5876): No heartbeat from core client for 30 sec - exiting 20:22:08 (5876): No heartbeat from core client for 30 sec - exiting 20:22:09 (5876): No heartbeat from core client for 30 sec - exiting 20:24:12 (5908): No heartbeat from core client for 30 sec - exiting 20:24:13 (5908): No heartbeat from core client for 30 sec - exiting 20:24:14 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:00 (5848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:01 (5848): No heartbeat from core client for 30 sec - exiting 10:03:02 (5848): No heartbeat from core client for 30 sec - exiting 10:03:03 (5848): No heartbeat from core client for 30 sec - exiting 10:03:04 (5848): No heartbeat from core client for 30 sec - exiting 10:03:05 (5848): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 09:53:01 (2456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:53:02 (2456): No heartbeat from core client for 30 sec - exiting 09:53:03 (2456): No heartbeat from core client for 30 sec - exiting 04:06:17 (4372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:18 (4372): No heartbeat from core client for 30 sec - exiting 04:06:19 (4372): No heartbeat from core client for 30 sec - exiting 04:06:20 (4372): No heartbeat from core client for 30 sec - exiting 04:06:21 (4372): No heartbeat from core client for 30 sec - exiting 04:06:22 (4372): No heartbeat from core client for 30 sec - exiting 04:06:23 (4372): No heartbeat from core client for 30 sec - exiting 04:06:24 (4372): No heartbeat from core client for 30 sec - exiting 12:33:04 (4688): No heartbeat from core client for 30 sec - exiting 12:33:05 (4688): No heartbeat from core client for 30 sec - exiting 12:33:06 (4688): No heartbeat from core client for 30 sec - exiting 12:33:07 (4688): No heartbeat from core client for 30 sec - exiting 12:33:08 (4688): No heartbeat from core client for 30 sec - exiting 12:33:09 (4688): No heartbeat from core client for 30 sec - exiting 12:33:10 (4688): No heartbeat from core client for 30 sec - exiting 12:33:11 (4688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:08:41 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:08:42 (4080): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 01:01:59 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:01 (5948): No heartbeat from core client for 30 sec - exiting 01:02:02 (5948): No heartbeat from core client for 30 sec - exiting 01:02:03 (5948): No heartbeat from core client for 30 sec - exiting 01:02:04 (5948): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 02:02:46 (2752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:47 (2752): No heartbeat from core client for 30 sec - exiting 02:02:48 (2752): No heartbeat from core client for 30 sec - exiting 02:02:49 (2752): No heartbeat from core client for 30 sec - exiting 02:02:50 (2752): No heartbeat from core client for 30 sec - exiting 02:13:11 (2136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:13 (2136): No heartbeat from core client for 30 sec - exiting 02:13:14 (2136): No heartbeat from core client for 30 sec - exiting 02:13:15 (2136): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4272, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 18:11:52 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:54 (4296): No heartbeat from core client for 30 sec - exiting 18:11:55 (4296): No heartbeat from core client for 30 sec - exiting 18:11:56 (4296): No heartbeat from core client for 30 sec - exiting 04:53:06 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:53:07 (2588): No heartbeat from core client for 30 sec - exiting 04:53:08 (2588): No heartbeat from core client for 30 sec - exiting 04:53:09 (2588): No heartbeat from core client for 30 sec - exiting 00:40:22 (2664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:40:23 (2664): No heartbeat from core client for 30 sec - exiting 00:40:24 (2664): No heartbeat from core client for 30 sec - exiting 00:40:25 (2664): No heartbeat from core client for 30 sec - exiting 00:40:26 (2664): No heartbeat from core client for 30 sec - exiting 00:40:27 (2664): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 00:48:16 (3004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:48:17 (3004): No heartbeat from core client for 30 sec - exiting 00:48:18 (3004): No heartbeat from core client for 30 sec - exiting 00:48:19 (3004): No heartbeat from core client for 30 sec - exiting 00:48:20 (3004): No heartbeat from core client for 30 sec - exiting 00:48:21 (3004): No heartbeat from core client for 30 sec - exiting 00:48:22 (3004): No heartbeat from core client for 30 sec - exiting 00:48:23 (3004): No heartbeat from core client for 30 sec - exiting 00:48:24 (3004): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ee2ko.dap4ah0 21:03:47 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:48 (6076): No heartbeat from core client for 30 sec - exiting 21:03:49 (6076): No heartbeat from core client for 30 sec - exiting 21:03:50 (6076): No heartbeat from core client for 30 sec - exiting 21:11:52 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:53 (5200): No heartbeat from core client for 30 sec - exiting 21:11:54 (5200): No heartbeat from core client for 30 sec - exiting 21:11:55 (5200): No heartbeat from core client for 30 sec - exiting 21:11:56 (5200): No heartbeat from core client for 30 sec - exiting 21:11:57 (5200): No heartbeat from core client for 30 sec - exiting 21:11:58 (5200): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1068, iMonCtr=1 Model crash detected, will try to restart... 07:44:18 (5376): No heartbeat from core client for 30 sec - exiting 07:44:19 (5376): No heartbeat from core client for 30 sec - exiting 07:44:20 (5376): No heartbeat from core client for 30 sec - exiting 07:44:21 (5376): No heartbeat from core client for 30 sec - exiting 07:44:22 (5376): No heartbeat from core client for 30 sec - exiting 07:44:23 (5376): No heartbeat from core client for 30 sec - exiting 07:44:24 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Aug 2013 09:29:45	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	1,036,800	4,181,579	4.0332
15 Aug 2013 18:59:37	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	1,010,880	4,069,348	4.0256
14 Aug 2013 17:38:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	984,960	3,961,352	4.0218
14 Aug 2013 17:38:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	959,040	3,862,875	4.0279
14 Aug 2013 17:38:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	933,120	3,761,844	4.0315
14 Aug 2013 17:38:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	907,200	3,661,129	4.0356
14 Aug 2013 17:38:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	881,280	3,557,965	4.0373
14 Aug 2013 17:38:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	855,360	3,449,804	4.0332
14 Aug 2013 17:38:31	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	829,440	3,346,604	4.0348
14 Aug 2013 17:38:31	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	803,520	3,243,037	4.0360
14 Aug 2013 17:38:31	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	777,600	3,132,820	4.0288
14 Aug 2013 17:38:31	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	751,680	3,014,447	4.0103
14 Aug 2013 17:38:31	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	725,760	2,899,676	3.9954
14 Aug 2013 17:38:31	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	699,840	2,792,322	3.9899
29 Jul 2013 13:22:46	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	673,920	2,682,060	3.9798
29 Jul 2013 13:22:46	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	648,000	2,562,089	3.9538
25 Jul 2013 23:33:44	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	622,080	2,443,498	3.9279
24 Jul 2013 10:43:32	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	596,160	2,323,204	3.8969
23 Jul 2013 21:54:09	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	570,240	2,201,457	3.8606
23 Jul 2013 20:59:39	1103902	15854650	hadcm3n_3ee2_2020_40_008393116_0	544,320	2,095,709	3.8501