Task 15920823

Name	hadcm3n_3jhf_2020_40_008364148_3
Workunit	8515007
Created	14 Aug 2013, 20:26:50 UTC
Sent	14 Aug 2013, 20:29:12 UTC
Report deadline	14 Nov 2013, 3:56:23 UTC
Received	11 Sep 2013, 14:56:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	178913
Run time	13 days 13 hours 59 min 18 sec
CPU time	13 days 0 hours 14 min 49 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	2.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> Zařízení nezná tento příkaz. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3jhfko.pjm1c10 Error converting file to netcdf: dataout/3jhfko.pim1c10 Error converting file to netcdf: dataout/3jhfko.pfm1c10 Error converting file to netcdf: dataout/3jhfka.phm1c10 Error converting file to netcdf: dataout/3jhfka.pgm1c10 Error converting file to netcdf: dataout/3jhfka.pem1c10 Error converting file to netcdf: dataout/3jhfka.pdm1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:15:28 (8564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:10:04 (6464): No heartbeat from core client for 30 sec - exiting 23:10:06 (6464): No heartbeat from core client for 30 sec - exiting 23:10:07 (6464): No heartbeat from core client for 30 sec - exiting 23:10:08 (6464): No heartbeat from core client for 30 sec - exiting 23:10:09 (6464): No heartbeat from core client for 30 sec - exiting 23:10:10 (6464): No heartbeat from core client for 30 sec - exiting 23:10:11 (6464): No heartbeat from core client for 30 sec - exiting 23:10:12 (6464): No heartbeat from core client for 30 sec - exiting 23:10:13 (6464): No heartbeat from core client for 30 sec - exiting 23:10:14 (6464): No heartbeat from core client for 30 sec - exiting 23:10:15 (6464): No heartbeat from core client for 30 sec - exiting 23:10:16 (6464): No heartbeat from core client for 30 sec - exiting 23:10:17 (6464): No heartbeat from core client for 30 sec - exiting 23:10:18 (6464): No heartbeat from core client for 30 sec - exiting 23:10:19 (6464): No heartbeat from core client for 30 sec - exiting 23:10:20 (6464): No heartbeat from core client for 30 sec - exiting 23:10:21 (6464): No heartbeat from core client for 30 sec - exiting 23:10:22 (6464): No heartbeat from core client for 30 sec - exiting 23:10:23 (6464): No heartbeat from core client for 30 sec - exiting 23:10:24 (6464): No heartbeat from core client for 30 sec - exiting 23:10:25 (6464): No heartbeat from core client for 30 sec - exiting 23:10:26 (6464): No heartbeat from core client for 30 sec - exiting 23:10:27 (6464): No heartbeat from core client for 30 sec - exiting 23:10:28 (6464): No heartbeat from core client for 30 sec - exiting 23:10:29 (6464): No heartbeat from core client for 30 sec - exiting 23:10:30 (6464): No heartbeat from core client for 30 sec - exiting 23:10:31 (6464): No heartbeat from core client for 30 sec - exiting 23:10:32 (6464): No heartbeat from core client for 30 sec - exiting 23:10:33 (6464): No heartbeat from core client for 30 sec - exiting 23:10:34 (6464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:49:34 (13692): No heartbeat from core client for 30 sec - exiting 09:49:35 (13692): No heartbeat from core client for 30 sec - exiting 09:49:36 (13692): No heartbeat from core client for 30 sec - exiting 09:49:37 (13692): No heartbeat from core client for 30 sec - exiting 09:49:38 (13692): No heartbeat from core client for 30 sec - exiting 09:49:39 (13692): No heartbeat from core client for 30 sec - exiting 09:49:40 (13692): No heartbeat from core client for 30 sec - exiting 09:49:41 (13692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:19:48 (16044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:07:36 (26740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:17 (37500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:41:15 (6500): No heartbeat from core client for 30 sec - exiting 12:41:17 (6500): No heartbeat from core client for 30 sec - exiting 12:41:18 (6500): No heartbeat from core client for 30 sec - exiting 12:41:19 (6500): No heartbeat from core client for 30 sec - exiting 12:41:20 (6500): No heartbeat from core client for 30 sec - exiting 12:41:21 (6500): No heartbeat from core client for 30 sec - exiting 12:41:22 (6500): No heartbeat from core client for 30 sec - exiting 12:41:23 (6500): No heartbeat from core client for 30 sec - exiting 12:41:24 (6500): No heartbeat from core client for 30 sec - exiting 12:41:25 (6500): No heartbeat from core client for 30 sec - exiting 12:41:26 (6500): No heartbeat from core client for 30 sec - exiting 12:41:27 (6500): No heartbeat from core client for 30 sec - exiting 12:41:28 (6500): No heartbeat from core client for 30 sec - exiting 12:41:29 (6500): No heartbeat from core client for 30 sec - exiting 12:41:30 (6500): No heartbeat from core client for 30 sec - exiting 12:41:31 (6500): No heartbeat from core client for 30 sec - exiting 12:41:32 (6500): No heartbeat from core client for 30 sec - exiting 12:41:33 (6500): No heartbeat from core client for 30 sec - exiting 12:41:34 (6500): No heartbeat from core client for 30 sec - exiting 12:41:35 (6500): No heartbeat from core client for 30 sec - exiting 12:41:36 (6500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:45 (5228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:38:05 (2780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Sep 2013 11:25:13	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	648,000	1,121,944	1.7314
08 Sep 2013 21:05:23	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	622,080	1,070,396	1.7207
07 Sep 2013 21:07:53	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	596,160	1,023,085	1.7161
06 Sep 2013 14:38:52	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	570,240	974,592	1.7091
05 Sep 2013 08:02:54	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	544,320	929,545	1.7077
04 Sep 2013 09:11:52	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	518,400	885,951	1.7090
02 Sep 2013 18:29:15	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	492,480	840,619	1.7069
01 Sep 2013 13:51:14	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	466,560	797,229	1.7087
31 Aug 2013 11:26:02	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	440,640	770,197	1.7479
30 Aug 2013 15:44:09	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	414,720	737,342	1.7779
29 Aug 2013 19:30:05	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	388,800	690,990	1.7772
28 Aug 2013 21:06:35	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	362,880	643,884	1.7744
28 Aug 2013 11:08:53	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	336,960	608,488	1.8058
27 Aug 2013 13:12:32	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	311,040	562,266	1.8077
26 Aug 2013 15:10:43	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	285,120	515,535	1.8081
25 Aug 2013 16:44:44	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	259,200	469,165	1.8101
23 Aug 2013 23:55:35	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	233,280	421,701	1.8077
23 Aug 2013 10:48:44	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	207,360	376,042	1.8135
21 Aug 2013 12:32:39	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	181,440	328,419	1.8101
20 Aug 2013 17:40:52	178913	15920823	hadcm3n_3jhf_2020_40_008364148_3	155,520	282,285	1.8151