Task 15132568

Name	hadcm3n_z9am_1880_40_008026224_4
Workunit	8181338
Created	16 Aug 2012, 18:40:13 UTC
Sent	16 Aug 2012, 18:41:14 UTC
Report deadline	16 Nov 2012, 2:08:25 UTC
Received	16 Nov 2012, 8:27:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1211878
Run time	25 days 3 hours 51 min 22 sec
CPU time	18 days 21 hours 36 min 16 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	2.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14004, selfPID=14004, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/z9amko.pja1c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pja1c10 Error: Input file: dataout/z9amko.pia1c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pia1c10 Error: Input file: dataout/z9amko.pfa1c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pfa1c10 Error: Input file: dataout/z9amka.pha1c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pha1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14216, selfPID=14216, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/z9amko.pja4c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pja4c10 Error: Input file: dataout/z9amko.pia4c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pia4c10 Error: Input file: dataout/z9amko.pfa4c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pfa4c10 Error: Input file: dataout/z9amka.pha4c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pha4c10 Error: Input file: dataout/z9amka.pga4c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pga4c10 Error: Input file: dataout/z9amka.pea4c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pea4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/z9amko.pja6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pja6c10 Error: Input file: dataout/z9amko.pia6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pia6c10 Error: Input file: dataout/z9amko.pfa6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amko.pfa6c10 Error: Input file: dataout/z9amka.pha6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pha6c10 Error: Input file: dataout/z9amka.pga6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pga6c10 Error: Input file: dataout/z9amka.pea6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pea6c10 Error: Input file: dataout/z9amka.pda6c10 is not a valid UM file. Error converting file to netcdf: dataout/z9amka.pda6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Nov 2012 07:05:40	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	1,010,880	1,835,563	1.8158
15 Nov 2012 15:15:08	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	984,960	1,788,169	1.8155
14 Nov 2012 22:37:58	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	959,040	1,739,599	1.8139
14 Nov 2012 06:38:47	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	933,120	1,691,551	1.8128
13 Nov 2012 14:54:15	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	907,200	1,644,207	1.8124
17 Oct 2012 00:38:28	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	881,280	1,595,432	1.8104
15 Oct 2012 10:55:59	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	855,360	1,547,865	1.8096
13 Oct 2012 18:36:11	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	829,440	1,500,673	1.8093
11 Oct 2012 22:58:22	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	803,520	1,453,697	1.8092
10 Oct 2012 06:50:53	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	777,600	1,405,821	1.8079
08 Oct 2012 18:11:17	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	751,680	1,358,511	1.8073
07 Oct 2012 06:34:15	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	725,760	1,312,546	1.8085
05 Oct 2012 07:02:10	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	699,840	1,266,144	1.8092
03 Oct 2012 03:45:36	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	673,920	1,219,149	1.8090
30 Sep 2012 19:51:41	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	648,000	1,172,103	1.8088
28 Sep 2012 16:33:23	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	622,080	1,124,963	1.8084
26 Sep 2012 05:31:56	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	596,160	1,076,742	1.8061
23 Sep 2012 23:42:50	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	570,240	1,029,425	1.8052
21 Sep 2012 17:14:01	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	544,320	982,377	1.8048
19 Sep 2012 13:19:12	1211878	15132568	hadcm3n_z9am_1880_40_008026224_4	518,400	935,038	1.8037