Task 16052395

Name	hadcm3n_3lb8_2020_40_008392873_2
Workunit	8543732
Created	1 Oct 2013, 16:45:59 UTC
Sent	1 Oct 2013, 17:03:55 UTC
Report deadline	1 Jan 2014, 0:31:06 UTC
Received	16 Feb 2014, 0:26:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1064436
Run time	19 days 22 hours 18 min 37 sec
CPU time	17 days 11 hours 26 min 52 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	3.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:11:25 (242): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:09 (6058): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:09 (6087): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:10 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:34:39 (243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(353,0xa049a540) malloc: * error for object 0x841200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133805) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1330, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133805) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1330, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133805) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1330, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133805) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1330, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133805) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1330, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 133805) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1330, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Oct 2013 16:56:05	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	1,010,880	1,498,321	1.4822
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	984,960	1,458,938	1.4812
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	959,040	1,419,581	1.4802
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	933,120	1,380,122	1.4790
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	907,200	1,340,783	1.4779
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	881,280	1,301,223	1.4765
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	855,360	1,261,745	1.4751
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	829,440	1,222,319	1.4737
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	803,520	1,182,901	1.4721
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	777,600	1,143,431	1.4705
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	751,680	1,104,040	1.4688
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	725,760	1,064,593	1.4669
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	699,840	1,025,118	1.4648
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	673,920	985,693	1.4626
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	648,000	946,235	1.4602
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	622,080	906,664	1.4575
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	596,160	867,291	1.4548
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	570,240	828,010	1.4520
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	544,320	788,778	1.4491
30 Oct 2013 16:01:49	1064436	16052395	hadcm3n_3lb8_2020_40_008392873_2	518,400	749,470	1.4457