Task 13606369

Name	hadcm3n_u2bd_1980_40_007538405_1
Workunit	7735637
Created	5 Nov 2011, 20:52:14 UTC
Sent	16 Nov 2011, 17:43:03 UTC
Report deadline	16 Feb 2012, 1:10:14 UTC
Received	27 Dec 2011, 1:58:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1161676
Run time	4 days 20 hours 52 min 18 sec
CPU time	4 days 14 hours 1 min 7 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	3.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2001804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2001800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2001804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2001800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2802204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2802200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2802204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2802200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(173,0xa0043540) malloc: * error for object 0x2802204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=177, selfPID=177, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3488,0xa0043540) malloc: * error for object 0x101de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136195) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=493, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136195) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=493, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136195) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=493, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136195) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=493, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136195) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=493, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136195) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=493, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Dec 2011 18:55:30	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	414,720	390,710	0.9421
25 Dec 2011 05:03:02	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	388,800	366,496	0.9426
24 Dec 2011 05:39:24	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	362,880	341,855	0.9421
23 Dec 2011 21:07:25	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	336,960	317,493	0.9422
23 Dec 2011 03:44:57	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	311,040	293,108	0.9423
22 Dec 2011 07:43:11	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	285,120	267,770	0.9391
21 Dec 2011 22:35:06	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	259,200	243,198	0.9383
21 Dec 2011 02:41:28	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	233,280	219,166	0.9395
20 Dec 2011 06:17:16	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	207,360	195,287	0.9418
19 Dec 2011 23:05:25	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	181,440	171,347	0.9444
19 Dec 2011 07:17:18	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	155,520	147,351	0.9475
16 Dec 2011 19:12:58	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	129,600	123,132	0.9501
15 Dec 2011 00:37:05	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	103,680	98,793	0.9529
13 Dec 2011 22:50:04	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	77,760	74,238	0.9547
12 Dec 2011 08:03:11	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	51,840	49,762	0.9599
11 Dec 2011 06:51:23	1161676	13606369	hadcm3n_u2bd_1980_40_007538405_1	25,920	24,789	0.9564