Task 13089614

Name	hadcm3n_y7q8_1900_40_007342874_1
Workunit	7540304
Created	6 Jul 2011, 13:14:27 UTC
Sent	6 Jul 2011, 18:06:28 UTC
Report deadline	6 Oct 2011, 1:33:39 UTC
Received	24 Jul 2011, 17:24:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1147765
Run time	6 days 6 hours 0 min 16 sec
CPU time	5 days 6 hours 23 min 22 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	3.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(772,0xa0b9e540) malloc: * error for object 0x1057404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(772,0xa0b9e540) malloc: * error for object 0x1057400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(772,0xa0b9e540) malloc: * error for object 0x83ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(772,0xa0b9e540) malloc: * error for object 0x858004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(772,0xa0b9e540) malloc: * error for object 0x858000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(772,0xa0b9e540) malloc: * error for object 0x83ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=774, selfPID=774, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(5015,0xa0b9e540) malloc: * error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x854604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x854604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1828e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(299,0xa0b9e540) malloc: * error for object 0x1829c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134790) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=553, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134790) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=553, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134790) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=553, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134790) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=553, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134790) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=553, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 134790) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=553, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 18:15:54	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	362,880	430,065	1.1851
25 Jul 2011 17:16:27	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	336,960	397,652	1.1801
25 Jul 2011 16:18:39	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	311,040	364,584	1.1721
25 Jul 2011 15:35:12	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	285,120	331,236	1.1617
25 Jul 2011 12:45:03	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	259,200	300,758	1.1603
25 Jul 2011 12:45:03	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	233,280	267,884	1.1483
25 Jul 2011 12:45:03	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	207,360	234,957	1.1331
25 Jul 2011 12:45:03	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	181,440	202,147	1.1141
25 Jul 2011 12:45:02	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	155,520	169,198	1.0880
25 Jul 2011 12:45:02	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	129,600	136,339	1.0520
10 Jul 2011 11:02:33	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	103,680	127,031	1.2252
09 Jul 2011 12:41:39	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	77,760	93,900	1.2076
08 Jul 2011 14:34:42	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	51,840	61,117	1.1790
07 Jul 2011 17:56:33	1147765	13089614	hadcm3n_y7q8_1900_40_007342874_1	25,920	28,445	1.0974