Task 15541219

Name	hadcm3n_ziqv_1920_40_008282165_0
Workunit	8433300
Created	14 Jan 2013, 1:40:51 UTC
Sent	14 Jan 2013, 1:41:02 UTC
Report deadline	15 Apr 2013, 9:08:13 UTC
Received	31 Jan 2013, 4:14:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1245245
Run time	4 days 18 hours 36 min 9 sec
CPU time	4 days 11 hours 18 min 59 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:28:41 (14721): No heartbeat from core client for 30 sec - exiting 21:28:42 (14721): No heartbeat from core client for 30 sec - exiting 21:28:43 (14721): No heartbeat from core client for 30 sec - exiting 21:28:44 (14721): No heartbeat from core client for 30 sec - exiting 21:28:45 (14721): No heartbeat from core client for 30 sec - exiting 21:28:46 (14721): No heartbeat from core client for 30 sec - exiting 21:28:47 (14721): No heartbeat from core client for 30 sec - exiting 21:28:48 (14721): No heartbeat from core client for 30 sec - exiting 21:28:49 (14721): No heartbeat from core client for 30 sec - exiting 21:28:50 (14721): No heartbeat from core client for 30 sec - exiting 21:28:51 (14721): No heartbeat from core client for 30 sec - exiting 21:28:52 (14721): No heartbeat from core client for 30 sec - exiting 21:28:53 (14721): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:20:19 (6157): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:20 (6157): No heartbeat from core client for 30 sec - exiting 08:20:21 (6157): No heartbeat from core client for 30 sec - exiting 08:20:22 (6157): No heartbeat from core client for 30 sec - exiting 08:20:23 (6157): No heartbeat from core client for 30 sec - exiting 08:20:24 (6157): No heartbeat from core client for 30 sec - exiting 08:20:25 (6157): No heartbeat from core client for 30 sec - exiting 08:20:26 (6157): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/ziqvko.pjd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pjd0c10 Error: Input file: dataout/ziqvko.pid0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pid0c10 Error: Input file: dataout/ziqvko.pfd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pfd0c10 Error: Input file: dataout/ziqvko.pcd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pcd0c10 Error: Input file: dataout/ziqvko.pbd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pbd0c10 Error: Input file: dataout/ziqvko.pad0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pad0c10 Error: Input file: dataout/ziqvka.phd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.phd0c10 Error: Input file: dataout/ziqvka.pgd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.pgd0c10 Error: Input file: dataout/ziqvka.ped0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.ped0c10 Error: Input file: dataout/ziqvka.pdd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.pdd0c10 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/ziqvko.pjd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pjd0c10 Error: Input file: dataout/ziqvko.pid0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pid0c10 Error: Input file: dataout/ziqvko.pfd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pfd0c10 Error: Input file: dataout/ziqvko.pcd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pcd0c10 Error: Input file: dataout/ziqvko.pbd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pbd0c10 Error: Input file: dataout/ziqvko.pad0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvko.pad0c10 Error: Input file: dataout/ziqvka.phd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.phd0c10 Error: Input file: dataout/ziqvka.pgd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.pgd0c10 Error: Input file: dataout/ziqvka.ped0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.ped0c10 Error: Input file: dataout/ziqvka.pdd0c10 is not a valid UM file. Error converting file to netcdf: dataout/ziqvka.pdd0c10 hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x238a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x238a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x4b89800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x3381a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x3390404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(83299,0xac2232c0) malloc: * error for object 0x3390400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_ziqv_1920_40_008282165/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jan 2013 20:22:23	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	259,200	386,378	1.4907
29 Jan 2013 16:20:43	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	233,280	345,804	1.4824
28 Jan 2013 02:18:28	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	207,360	305,642	1.4740
27 Jan 2013 02:19:46	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	181,440	268,224	1.4783
26 Jan 2013 17:58:58	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	155,520	234,247	1.5062
19 Jan 2013 19:39:59	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	129,600	194,450	1.5004
18 Jan 2013 16:21:48	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	103,680	155,316	1.4980
17 Jan 2013 15:52:04	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	77,760	115,865	1.4900
15 Jan 2013 15:35:55	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	51,840	74,902	1.4449
15 Jan 2013 05:07:45	1245245	15541219	hadcm3n_ziqv_1920_40_008282165_0	25,920	38,374	1.4805