Task 15463520

Name	hadcm3n_zmtu_1920_40_008254367_1
Workunit	8409491
Created	26 Nov 2012, 19:24:17 UTC
Sent	26 Nov 2012, 19:24:23 UTC
Report deadline	26 Feb 2013, 2:51:34 UTC
Received	15 Dec 2012, 20:04:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1176125
Run time	14 days 9 hours 53 min 10 sec
CPU time	13 days 9 hours 35 min 35 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x9f0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x31ecc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x31ecc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x9f0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x31ecc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x3208204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x3208200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x4205204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x11fd204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x11fd200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x31ecc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68913,0xac72d2c0) malloc: * error for object 0x49de804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(71287,0xac72d2c0) malloc: * error for object 0x3be6a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zmtu_1920_40_008254367/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Dec 2012 17:46:28	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	518,400	1,157,767	2.2333
14 Dec 2012 08:04:19	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	492,480	1,101,106	2.2358
14 Dec 2012 05:41:49	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	466,560	1,045,718	2.2413
14 Dec 2012 05:41:49	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	440,640	988,065	2.2423
14 Dec 2012 05:41:49	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	414,720	928,859	2.2397
14 Dec 2012 05:41:49	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	388,800	869,202	2.2356
14 Dec 2012 05:41:49	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	362,880	809,440	2.2306
07 Dec 2012 23:09:19	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	336,960	749,761	2.2251
07 Dec 2012 04:51:20	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	311,040	689,930	2.2181
06 Dec 2012 11:13:48	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	285,120	630,467	2.2112
05 Dec 2012 11:19:56	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	259,200	571,354	2.2043
04 Dec 2012 14:38:32	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	233,280	512,590	2.1973
03 Dec 2012 21:34:09	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	207,360	454,356	2.1911
02 Dec 2012 22:47:22	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	181,440	401,755	2.2143
02 Dec 2012 08:28:34	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	155,520	350,254	2.2521
01 Dec 2012 14:37:35	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	129,600	291,380	2.2483
30 Nov 2012 21:28:18	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	103,680	232,616	2.2436
29 Nov 2012 21:54:19	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	77,760	173,762	2.2346
28 Nov 2012 21:54:44	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	51,840	118,438	2.2847
27 Nov 2012 22:44:13	1176125	15463520	hadcm3n_zmtu_1920_40_008254367_1	25,920	59,379	2.2909