Task 14280602

Name	hadcm3n_yfyd_1980_40_007831681_3
Workunit	7986793
Created	17 Mar 2012, 22:49:45 UTC
Sent	17 Mar 2012, 22:50:00 UTC
Report deadline	17 Jun 2012, 6:17:11 UTC
Received	9 Apr 2012, 20:08:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1173904
Run time	10 days 21 hours 18 min 51 sec
CPU time	10 days 12 hours 49 min 45 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	3.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:32:01 (89706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0x332ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0x332ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0xb23e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0x4b23004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51573,0xac8912c0) malloc: * error for object 0x4b23000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(84402,0xac8912c0) malloc: * error for object 0x4092a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84402,0xac8912c0) malloc: * error for object 0x4092a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84402,0xac8912c0) malloc: * error for object 0x3097a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84402,0xac8912c0) malloc: * error for object 0x3097a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84402,0xac8912c0) malloc: * error for object 0x3093004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.3 build 11D50b Thu Apr 5 04:41:37 2012 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=84408, selfPID=84408, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x3075204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x2891204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x2892004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x2892000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x2892004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x2892000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14348,0xac8912c0) malloc: * error for object 0x2892000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:46:03 (34535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Apr 2012 09:45:30	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	855,360	902,129	1.0547
09 Apr 2012 02:02:07	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	829,440	874,839	1.0547
08 Apr 2012 18:18:48	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	803,520	847,543	1.0548
08 Apr 2012 08:33:27	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	777,600	820,318	1.0549
08 Apr 2012 00:04:20	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	751,680	792,977	1.0549
07 Apr 2012 16:20:37	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	725,760	765,717	1.0551
07 Apr 2012 07:15:20	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	699,840	738,484	1.0552
06 Apr 2012 23:47:15	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	673,920	711,152	1.0552
06 Apr 2012 15:08:10	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	648,000	684,132	1.0558
06 Apr 2012 06:59:08	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	622,080	656,737	1.0557
05 Apr 2012 22:29:56	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	596,160	629,404	1.0558
05 Apr 2012 14:42:41	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	570,240	602,012	1.0557
05 Apr 2012 06:08:00	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	544,320	574,872	1.0561
04 Apr 2012 19:43:33	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	518,400	547,581	1.0563
04 Apr 2012 12:00:40	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	492,480	520,251	1.0564
04 Apr 2012 03:51:52	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	466,560	492,927	1.0565
03 Apr 2012 20:02:17	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	440,640	465,761	1.0570
03 Apr 2012 11:52:41	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	414,720	438,557	1.0575
03 Apr 2012 02:32:47	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	388,800	411,158	1.0575
02 Apr 2012 18:47:40	1173904	14280602	hadcm3n_yfyd_1980_40_007831681_3	362,880	383,761	1.0575