Task 15842107

Name	hadcm3n_o204_1940_40_008381931_1
Workunit	8532790
Created	14 Jun 2013, 0:49:36 UTC
Sent	14 Jun 2013, 1:22:11 UTC
Report deadline	13 Sep 2013, 8:49:22 UTC
Received	19 Oct 2013, 7:45:53 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1213910
Run time	89 days 3 hours 3 min
CPU time	83 days 9 hours 2 min 53 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	0.58 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 08:29:43 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:59:38 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:22:26 (8904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:36 (2684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:06 (8732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:08 (8732): No heartbeat from core client for 30 sec - exiting 02:49:09 (8732): No heartbeat from core client for 30 sec - exiting 02:49:10 (8732): No heartbeat from core client for 30 sec - exiting 02:49:11 (8732): No heartbeat from core client for 30 sec - exiting 02:49:12 (8732): No heartbeat from core client for 30 sec - exiting 02:49:13 (8732): No heartbeat from core client for 30 sec - exiting 02:49:14 (8732): No heartbeat from core client for 30 sec - exiting 02:49:15 (8732): No heartbeat from core client for 30 sec - exiting 02:49:16 (8732): No heartbeat from core client for 30 sec - exiting 02:49:17 (8732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:16:02 (1388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:17 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:34 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:35 (1716): No heartbeat from core client for 30 sec - exiting 17:20:36 (1716): No heartbeat from core client for 30 sec - exiting 17:20:37 (1716): No heartbeat from core client for 30 sec - exiting 17:20:38 (1716): No heartbeat from core client for 30 sec - exiting 17:20:39 (1716): No heartbeat from core client for 30 sec - exiting 17:20:40 (1716): No heartbeat from core client for 30 sec - exiting 17:20:42 (1716): No heartbeat from core client for 30 sec - exiting 17:20:43 (1716): No heartbeat from core client for 30 sec - exiting 17:20:44 (1716): No heartbeat from core client for 30 sec - exiting 17:20:45 (1716): No heartbeat from core client for 30 sec - exiting 17:20:46 (1716): No heartbeat from core client for 30 sec - exiting 17:20:47 (1716): No heartbeat from core client for 30 sec - exiting 17:20:48 (1716): No heartbeat from core client for 30 sec - exiting 17:20:49 (1716): No heartbeat from core client for 30 sec - exiting 17:20:50 (1716): No heartbeat from core client for 30 sec - exiting 17:20:51 (1716): No heartbeat from core client for 30 sec - exiting 17:20:52 (1716): No heartbeat from core client for 30 sec - exiting 17:20:54 (1716): No heartbeat from core client for 30 sec - exiting 17:20:55 (1716): No heartbeat from core client for 30 sec - exiting 17:20:56 (1716): No heartbeat from core client for 30 sec - exiting 17:20:57 (1716): No heartbeat from core client for 30 sec - exiting 17:20:58 (1716): No heartbeat from core client for 30 sec - exiting 17:20:59 (1716): No heartbeat from core client for 30 sec - exiting 17:21:00 (1716): No heartbeat from core client for 30 sec - exiting 17:21:01 (1716): No heartbeat from core client for 30 sec - exiting 17:21:02 (1716): No heartbeat from core client for 30 sec - exiting 17:21:03 (1716): No heartbeat from core client for 30 sec - exiting 17:21:04 (1716): No heartbeat from core client for 30 sec - exiting 17:21:06 (1716): No heartbeat from core client for 30 sec - exiting 17:21:07 (1716): No heartbeat from core client for 30 sec - exiting 17:21:08 (1716): No heartbeat from core client for 30 sec - exiting 17:21:09 (1716): No heartbeat from core client for 30 sec - exiting 17:21:10 (1716): No heartbeat from core client for 30 sec - exiting 17:21:11 (1716): No heartbeat from core client for 30 sec - exiting 17:21:12 (1716): No heartbeat from core client for 30 sec - exiting 21:25:39 (1804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o204ko.pjf6c10 Error converting file to netcdf: dataout/o204ko.pif6c10 Error converting file to netcdf: dataout/o204ko.pff6c10 Error converting file to netcdf: dataout/o204ka.phf6c10 Error converting file to netcdf: dataout/o204ka.pgf6c10 Error converting file to netcdf: dataout/o204ka.pef6c10 Error converting file to netcdf: dataout/o204ka.pdf6c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:09:23 (1296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:18:52 (6816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:58 (6816): No heartbeat from core client for 30 sec - exiting 18:18:59 (6816): No heartbeat from core client for 30 sec - exiting 19:56:57 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:20 (1772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:55:56 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:56 (7296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:23 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:51 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:54 (7476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:53:34 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:32 (6508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:56:43 (7176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:54 (7632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:52:49 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:52:50 (3460): No heartbeat from core client for 30 sec - exiting 06:52:51 (3460): No heartbeat from core client for 30 sec - exiting 06:52:52 (3460): No heartbeat from core client for 30 sec - exiting 06:52:53 (3460): No heartbeat from core client for 30 sec - exiting 06:52:54 (3460): No heartbeat from core client for 30 sec - exiting 06:52:55 (3460): No heartbeat from core client for 30 sec - exiting 06:52:56 (3460): No heartbeat from core client for 30 sec - exiting 06:52:57 (3460): No heartbeat from core client for 30 sec - exiting 06:52:58 (3460): No heartbeat from core client for 30 sec - exiting 06:52:59 (3460): No heartbeat from core client for 30 sec - exiting 12:53:26 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:27 (3364): No heartbeat from core client for 30 sec - exiting 12:53:28 (3364): No heartbeat from core client for 30 sec - exiting 12:53:29 (3364): No heartbeat from core client for 30 sec - exiting 12:53:30 (3364): No heartbeat from core client for 30 sec - exiting 12:53:31 (3364): No heartbeat from core client for 30 sec - exiting 12:53:32 (3364): No heartbeat from core client for 30 sec - exiting 12:53:33 (3364): No heartbeat from core client for 30 sec - exiting 12:53:34 (3364): No heartbeat from core client for 30 sec - exiting 12:53:35 (3364): No heartbeat from core client for 30 sec - exiting 12:53:36 (3364): No heartbeat from core client for 30 sec - exiting 13:12:46 (1556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:47 (1556): No heartbeat from core client for 30 sec - exiting 13:12:48 (1556): No heartbeat from core client for 30 sec - exiting 13:12:49 (1556): No heartbeat from core client for 30 sec - exiting 13:12:50 (1556): No heartbeat from core client for 30 sec - exiting 13:12:51 (1556): No heartbeat from core client for 30 sec - exiting 13:12:52 (1556): No heartbeat from core client for 30 sec - exiting 13:12:53 (1556): No heartbeat from core client for 30 sec - exiting 13:12:54 (1556): No heartbeat from core client for 30 sec - exiting 13:12:55 (1556): No heartbeat from core client for 30 sec - exiting 13:12:56 (1556): No heartbeat from core client for 30 sec - exiting 14:34:26 (552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:34:27 (552): No heartbeat from core client for 30 sec - exiting 14:34:28 (552): No heartbeat from core client for 30 sec - exiting 14:34:29 (552): No heartbeat from core client for 30 sec - exiting 14:34:30 (552): No heartbeat from core client for 30 sec - exiting 14:34:31 (552): No heartbeat from core client for 30 sec - exiting 14:34:32 (552): No heartbeat from core client for 30 sec - exiting 14:34:33 (552): No heartbeat from core client for 30 sec - exiting 14:34:34 (552): No heartbeat from core client for 30 sec - exiting 14:34:35 (552): No heartbeat from core client for 30 sec - exiting 14:34:36 (552): No heartbeat from core client for 30 sec - exiting 04:53:18 (8072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:53:19 (8072): No heartbeat from core client for 30 sec - exiting 04:53:20 (8072): No heartbeat from core client for 30 sec - exiting 04:53:21 (8072): No heartbeat from core client for 30 sec - exiting 04:53:22 (8072): No heartbeat from core client for 30 sec - exiting 04:53:23 (8072): No heartbeat from core client for 30 sec - exiting 04:53:24 (8072): No heartbeat from core client for 30 sec - exiting 04:53:25 (8072): No heartbeat from core client for 30 sec - exiting 04:53:26 (8072): No heartbeat from core client for 30 sec - exiting 04:53:27 (8072): No heartbeat from core client for 30 sec - exiting 04:53:28 (8072): No heartbeat from core client for 30 sec - exiting 07:48:16 (7232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:48:17 (7232): No heartbeat from core client for 30 sec - exiting 07:48:18 (7232): No heartbeat from core client for 30 sec - exiting 07:48:19 (7232): No heartbeat from core client for 30 sec - exiting 07:48:20 (7232): No heartbeat from core client for 30 sec - exiting 07:48:21 (7232): No heartbeat from core client for 30 sec - exiting 07:48:22 (7232): No heartbeat from core client for 30 sec - exiting 07:48:23 (7232): No heartbeat from core client for 30 sec - exiting 07:48:24 (7232): No heartbeat from core client for 30 sec - exiting 07:48:25 (7232): No heartbeat from core client for 30 sec - exiting 07:48:26 (7232): No heartbeat from core client for 30 sec - exiting 21:34:26 (796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA CPDN Monitor - Quit request from BOINC... 03:05:32 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6680, iMonCtr=1 Model crash detected, will try to restart... 01:48:57 (2712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:36:50 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:35 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:53 (2448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Oct 2013 06:29:46	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	1,036,800	7,200,661	6.9451
17 Oct 2013 22:15:43	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	1,010,880	6,995,601	6.9203
17 Oct 2013 22:15:43	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	984,960	6,794,713	6.8985
09 Oct 2013 20:38:57	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	959,040	6,584,678	6.8659
09 Oct 2013 20:38:57	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	933,120	6,375,881	6.8329
04 Oct 2013 03:33:33	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	907,200	6,167,553	6.7984
02 Oct 2013 19:24:33	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	881,280	5,959,010	6.7618
02 Oct 2013 19:24:33	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	855,360	5,755,743	6.7290
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	829,440	5,664,573	6.8294
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	803,520	5,534,187	6.8874
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	777,600	5,403,827	6.9494
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	751,680	5,272,975	7.0149
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	725,760	5,128,124	7.0659
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	699,840	4,915,403	7.0236
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	673,920	4,698,253	6.9715
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	648,000	4,485,010	6.9213
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	622,080	4,282,788	6.8846
26 Sep 2013 23:54:30	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	596,160	4,081,373	6.8461
06 Sep 2013 16:19:44	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	570,240	3,882,253	6.8081
03 Sep 2013 22:11:52	1213910	15842107	hadcm3n_o204_1940_40_008381931_1	544,320	3,672,406	6.7468