Task 15443417

Name	hadcm3n_z86a_1880_40_008246886_1
Workunit	8402010
Created	21 Nov 2012, 4:09:37 UTC
Sent	21 Nov 2012, 4:09:43 UTC
Report deadline	20 Feb 2013, 11:36:54 UTC
Received	3 Dec 2012, 0:37:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	859190
Run time	6 days 17 hours 8 min 30 sec
CPU time	6 days 8 hours 10 min 58 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	1.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 10:47:48 (4316): No heartbeat from core client for 30 sec - exiting 10:47:49 (4316): No heartbeat from core client for 30 sec - exiting 10:47:50 (4316): No heartbeat from core client for 30 sec - exiting 10:47:51 (4316): No heartbeat from core client for 30 sec - exiting 10:47:53 (4316): No heartbeat from core client for 30 sec - exiting 10:47:54 (4316): No heartbeat from core client for 30 sec - exiting 10:47:55 (4316): No heartbeat from core client for 30 sec - exiting 10:47:56 (4316): No heartbeat from core client for 30 sec - exiting 10:47:57 (4316): No heartbeat from core client for 30 sec - exiting 10:47:58 (4316): No heartbeat from core client for 30 sec - exiting 10:47:59 (4316): No heartbeat from core client for 30 sec - exiting 10:48:00 (4316): No heartbeat from core client for 30 sec - exiting 10:48:01 (4316): No heartbeat from core client for 30 sec - exiting 10:48:02 (4316): No heartbeat from core client for 30 sec - exiting 10:48:03 (4316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:04 (4316): No heartbeat from core client for 30 sec - exiting Signal 1 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish 13:47:30 (4273): No heartbeat from core client for 30 sec - exiting 13:47:32 (4273): No heartbeat from core client for 30 sec - exiting 13:47:33 (4273): No heartbeat from core client for 30 sec - exiting 13:47:34 (4273): No heartbeat from core client for 30 sec - exiting 13:47:35 (4273): No heartbeat from core client for 30 sec - exiting 13:47:36 (4273): No heartbeat from core client for 30 sec - exiting 13:47:37 (4273): No heartbeat from core client for 30 sec - exiting 13:47:38 (4273): No heartbeat from core client for 30 sec - exiting 13:47:39 (4273): No heartbeat from core client for 30 sec - exiting 13:47:40 (4273): No heartbeat from core client for 30 sec - exiting 13:47:41 (4273): No heartbeat from core client for 30 sec - exiting 13:47:45 (4273): No heartbeat from core client for 30 sec - exiting 13:47:46 (4273): No heartbeat from core client for 30 sec - exiting 13:47:47 (4273): No heartbeat from core client for 30 sec - exiting 13:47:48 (4273): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:50 (4566): No heartbeat from core client for 30 sec - exiting 10:32:51 (4566): No heartbeat from core client for 30 sec - exiting 10:32:52 (4566): No heartbeat from core client for 30 sec - exiting 10:32:53 (4566): No heartbeat from core client for 30 sec - exiting 10:32:54 (4566): No heartbeat from core client for 30 sec - exiting 10:32:55 (4566): No heartbeat from core client for 30 sec - exiting 10:32:56 (4566): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish 10:34:22 (3809): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28383, iMonCtr=1 07:06:30 (4321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 09:21:10 (4637): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/z86ako.pj88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86ako.pj88c10 Error: Input file: dataout/z86ako.pi88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86ako.pi88c10 Error: Input file: dataout/z86ako.pf88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86ako.pf88c10 Error: Input file: dataout/z86aka.ph88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86aka.ph88c10 Error: Input file: dataout/z86aka.pg88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86aka.pg88c10 Error: Input file: dataout/z86aka.pe88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86aka.pe88c10 Error: Input file: dataout/z86aka.pd88c10 is not a valid UM file. Error converting file to netcdf: dataout/z86aka.pd88c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 10:33:23 (4789): No heartbeat from core client for 30 sec - exiting 10:33:24 (4789): No heartbeat from core client for 30 sec - exiting 10:33:25 (4789): No heartbeat from core client for 30 sec - exiting 10:33:26 (4789): No heartbeat from core client for 30 sec - exiting 10:33:27 (4789): No heartbeat from core client for 30 sec - exiting 10:33:28 (4789): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish 11:13:04 (4209): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:32:36 (3581): No heartbeat from core client for 30 sec - exiting 10:32:38 (3581): No heartbeat from core client for 30 sec - exiting 10:32:39 (3581): No heartbeat from core client for 30 sec - exiting 10:32:40 (3581): No heartbeat from core client for 30 sec - exiting 10:32:41 (3581): No heartbeat from core client for 30 sec - exiting 10:32:42 (3581): No heartbeat from core client for 30 sec - exiting 10:32:43 (3581): No heartbeat from core client for 30 sec - exiting 10:32:44 (3581): No heartbeat from core client for 30 sec - exiting 10:32:45 (3581): No heartbeat from core client for 30 sec - exiting 10:32:46 (3581): No heartbeat from core client for 30 sec - exiting 10:32:47 (3581): No heartbeat from core client for 30 sec - exiting 10:32:48 (3581): No heartbeat from core client for 30 sec - exiting 10:32:49 (3581): No heartbeat from core client for 30 sec - exiting 10:32:50 (3581): No heartbeat from core client for 30 sec - exiting 10:32:51 (3581): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:52 (3581): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 1 received, exiting... Signal 15 received, exiting... Called boinc_finish 07:23:12 (4203): No heartbeat from core client for 30 sec - exiting 07:23:14 (4203): No heartbeat from core client for 30 sec - exiting 07:23:15 (4203): No heartbeat from core client for 30 sec - exiting 07:23:16 (4203): No heartbeat from core client for 30 sec - exiting 07:23:17 (4203): No heartbeat from core client for 30 sec - exiting 07:23:18 (4203): No heartbeat from core client for 30 sec - exiting 07:23:19 (4203): No heartbeat from core client for 30 sec - exiting 07:23:20 (4203): No heartbeat from core client for 30 sec - exiting 07:23:21 (4203): No heartbeat from core client for 30 sec - exiting 07:23:22 (4203): No heartbeat from core client for 30 sec - exiting 07:23:23 (4203): No heartbeat from core client for 30 sec - exiting 07:23:24 (4203): No heartbeat from core client for 30 sec - exiting 07:23:25 (4203): No heartbeat from core client for 30 sec - exiting 07:23:26 (4203): No heartbeat from core client for 30 sec - exiting 07:23:27 (4203): No heartbeat from core client for 30 sec - exiting 07:23:28 (4203): No heartbeat from core client for 30 sec - exiting 07:23:29 (4203): No heartbeat from core client for 30 sec - exiting 07:23:30 (4203): No heartbeat from core client for 30 sec - exiting 07:23:31 (4203): No heartbeat from core client for 30 sec - exiting 07:23:32 (4203): No heartbeat from core client for 30 sec - exiting 07:23:33 (4203): No heartbeat from core client for 30 sec - exiting 07:23:34 (4203): No heartbeat from core client for 30 sec - exiting 07:23:35 (4203): No heartbeat from core client for 30 sec - exiting 07:23:36 (4203): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 1 received, exiting... Signal 15 received, exiting... Called boinc_finish 03:50:31 (4536): No heartbeat from core client for 30 sec - exiting 03:50:32 (4536): No heartbeat from core client for 30 sec - exiting 03:50:33 (4536): No heartbeat from core client for 30 sec - exiting 03:50:34 (4536): No heartbeat from core client for 30 sec - exiting 03:50:35 (4536): No heartbeat from core client for 30 sec - exiting 03:50:36 (4536): No heartbeat from core client for 30 sec - exiting 03:50:37 (4536): No heartbeat from core client for 30 sec - exiting 03:50:38 (4536): No heartbeat from core client for 30 sec - exiting 03:50:39 (4536): No heartbeat from core client for 30 sec - exiting 03:50:40 (4536): No heartbeat from core client for 30 sec - exiting 03:50:41 (4536): No heartbeat from core client for 30 sec - exiting 03:50:42 (4536): No heartbeat from core client for 30 sec - exiting 03:50:43 (4536): No heartbeat from core client for 30 sec - exiting 03:50:44 (4536): No heartbeat from core client for 30 sec - exiting 03:50:45 (4536): No heartbeat from core client for 30 sec - exiting 03:50:46 (4536): No heartbeat from core client for 30 sec - exiting 03:50:47 (4536): No heartbeat from core client for 30 sec - exiting 03:50:48 (4536): No heartbeat from core client for 30 sec - exiting 03:50:49 (4536): No heartbeat from core client for 30 sec - exiting 03:50:50 (4536): No heartbeat from core client for 30 sec - exiting 03:50:51 (4536): No heartbeat from core client for 30 sec - exiting 03:50:52 (4536): No heartbeat from core client for 30 sec - exiting 03:50:53 (4536): No heartbeat from core client for 30 sec - exiting 03:50:54 (4536): No heartbeat from core client for 30 sec - exiting 03:50:56 (4536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Dec 2012 11:29:08	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	466,560	553,352	1.1860
02 Dec 2012 01:22:14	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	440,640	522,503	1.1858
01 Dec 2012 16:43:00	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	414,720	491,644	1.1855
01 Dec 2012 08:16:08	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	388,800	460,731	1.1850
30 Nov 2012 23:14:06	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	362,880	429,915	1.1847
30 Nov 2012 14:28:59	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	336,960	399,059	1.1843
29 Nov 2012 23:19:37	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	311,040	368,307	1.1841
29 Nov 2012 02:56:14	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	285,120	337,548	1.1839
27 Nov 2012 20:07:16	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	259,200	306,748	1.1834
27 Nov 2012 01:22:08	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	233,280	275,956	1.1829
25 Nov 2012 18:38:19	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	207,360	245,330	1.1831
25 Nov 2012 03:44:21	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	181,440	214,669	1.1831
24 Nov 2012 17:15:24	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	155,520	183,969	1.1829
24 Nov 2012 08:33:23	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	129,600	153,282	1.1827
23 Nov 2012 23:46:21	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	103,680	122,683	1.1833
23 Nov 2012 02:34:19	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	77,760	92,000	1.1831
22 Nov 2012 18:10:23	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	51,840	61,284	1.1822
21 Nov 2012 23:55:55	859190	15443417	hadcm3n_z86a_1880_40_008246886_1	25,920	30,698	1.1843