Task 12834547

Name	hadcm3n_p7i4_1900_40_007227308_1
Workunit	7425548
Created	26 Apr 2011, 15:39:47 UTC
Sent	26 Apr 2011, 18:47:51 UTC
Report deadline	27 Jul 2011, 2:15:02 UTC
Received	3 Nov 2011, 23:33:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1047699
Run time	8 days 10 hours 16 min 49 sec
CPU time	7 days 0 hours 31 min 22 sec
Validate state	Invalid
Credit	6,531.84
Device peak FLOPS	3.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x83ec04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x83ec00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x2843004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x2843000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x2843004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x2843000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x102c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95456,0xa0030540) malloc: * error for object 0x102c600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K540 Mon Jul 4 09:57:24 2011 Thread 0 Crashed: Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=95461, selfPID=95461, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x832c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x2828604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x182f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x182f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(13812,0xa0030540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x2824604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x103ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21137,0xa0030540) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(39273,0xa0030540) malloc: * error for object 0x1802604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39273,0xa0030540) malloc: * error for object 0x1802600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39273,0xa0030540) malloc: * error for object 0x82de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43060,0xa0030540) malloc: * error for object 0x1810400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133015) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70578, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133015) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70578, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133015) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70578, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133015) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70578, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133015) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70578, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133015) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=70578, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2011 07:23:25	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	544,320	599,295	1.1010
04 Jul 2011 14:02:11	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	518,400	570,257	1.1000
02 Jul 2011 09:31:31	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	492,480	541,054	1.0986
01 Jul 2011 00:04:43	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	466,560	517,985	1.1102
28 Jun 2011 12:26:17	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	440,640	488,479	1.1086
21 Jun 2011 19:40:16	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	414,720	467,163	1.1265
20 Jun 2011 08:35:10	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	388,800	437,839	1.1261
17 Jun 2011 08:35:32	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	362,880	408,565	1.1259
10 Jun 2011 00:23:28	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	336,960	379,461	1.1261
07 Jun 2011 09:50:09	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	311,040	350,266	1.1261
05 Jun 2011 08:05:35	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	285,120	321,082	1.1261
02 Jun 2011 08:17:50	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	259,200	291,845	1.1259
01 Jun 2011 23:28:03	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	233,280	262,584	1.1256
30 May 2011 20:29:19	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	207,360	233,511	1.1261
28 May 2011 17:06:57	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	181,440	204,440	1.1268
20 May 2011 14:59:08	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	155,520	175,319	1.1273
16 May 2011 17:13:53	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	129,600	146,120	1.1275
09 May 2011 08:26:03	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	103,680	116,931	1.1278
07 May 2011 08:27:22	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	77,760	87,684	1.1276
05 May 2011 21:07:21	1047699	12834547	hadcm3n_p7i4_1900_40_007227308_1	51,840	58,457	1.1276