Task 13358146

Name	hadcm3n_o55h_1940_40_007446504_2
Workunit	7644007
Created	9 Sep 2011, 16:53:07 UTC
Sent	16 Sep 2011, 18:31:05 UTC
Report deadline	17 Dec 2011, 1:58:16 UTC
Received	29 Sep 2011, 5:45:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	615938
Run time	9 days 0 hours 58 min 54 sec
CPU time	8 days 19 hours 31 min 42 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:15:16 (43425): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:59:05 (47140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:01:59 (51571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x10eea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x10eea00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x80eea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x40e1a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x38c4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x8c3c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(80943,0xac7f82c0) malloc: * error for object 0x8c3c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:36:00 (280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:01 (280): No heartbeat from core client for 30 sec - exiting 11:36:02 (280): No heartbeat from core client for 30 sec - exiting 11:36:03 (280): No heartbeat from core client for 30 sec - exiting 11:36:04 (280): No heartbeat from core client for 30 sec - exiting 11:36:05 (280): No heartbeat from core client for 30 sec - exiting 11:36:06 (280): No heartbeat from core client for 30 sec - exiting 11:36:07 (280): No heartbeat from core client for 30 sec - exiting 11:36:08 (280): No heartbeat from core client for 30 sec - exiting 11:36:09 (280): No heartbeat from core client for 30 sec - exiting 11:36:10 (280): No heartbeat from core client for 30 sec - exiting 11:36:11 (280): No heartbeat from core client for 30 sec - exiting 11:36:12 (280): No heartbeat from core client for 30 sec - exiting 11:36:13 (280): No heartbeat from core client for 30 sec - exiting 11:36:14 (280): No heartbeat from core client for 30 sec - exiting 11:36:15 (280): No heartbeat from core client for 30 sec - exiting 11:36:16 (280): No heartbeat from core client for 30 sec - exiting 11:36:17 (280): No heartbeat from core client for 30 sec - exiting 11:36:18 (280): No heartbeat from core client for 30 sec - exiting 11:36:19 (280): No heartbeat from core client for 30 sec - exiting 11:36:20 (280): No heartbeat from core client for 30 sec - exiting 11:36:21 (280): No heartbeat from core client for 30 sec - exiting 11:36:22 (280): No heartbeat from core client for 30 sec - exiting 11:36:23 (280): No heartbeat from core client for 30 sec - exiting 11:36:24 (280): No heartbeat from core client for 30 sec - exiting 11:36:25 (280): No heartbeat from core client for 30 sec - exiting 11:36:26 (280): No heartbeat from core client for 30 sec - exiting 11:36:27 (280): No heartbeat from core client for 30 sec - exiting 11:36:28 (280): No heartbeat from core client for 30 sec - exiting 11:36:29 (280): No heartbeat from core client for 30 sec - exiting 11:36:30 (280): No heartbeat from core client for 30 sec - exiting 11:36:31 (280): No heartbeat from core client for 30 sec - exiting 11:36:32 (280): No heartbeat from core client for 30 sec - exiting 11:36:33 (280): No heartbeat from core client for 30 sec - exiting 11:36:34 (280): No heartbeat from core client for 30 sec - exiting 11:36:35 (280): No heartbeat from core client for 30 sec - exiting 11:36:36 (280): No heartbeat from core client for 30 sec - exiting 11:36:37 (280): No heartbeat from core client for 30 sec - exiting 11:36:38 (280): No heartbeat from core client for 30 sec - exiting 11:36:39 (280): No heartbeat from core client for 30 sec - exiting 11:36:40 (280): No heartbeat from core client for 30 sec - exiting 11:36:41 (280): No heartbeat from core client for 30 sec - exiting 11:36:42 (280): No heartbeat from core client for 30 sec - exiting 11:36:43 (280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:20:44 (13433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:45 (13433): No heartbeat from core client for 30 sec - exiting 19:20:46 (13433): No heartbeat from core client for 30 sec - exiting 19:20:47 (13433): No heartbeat from core client for 30 sec - exiting 19:20:48 (13433): No heartbeat from core client for 30 sec - exiting 19:20:49 (13433): No heartbeat from core client for 30 sec - exiting 19:20:50 (13433): No heartbeat from core client for 30 sec - exiting 19:20:51 (13433): No heartbeat from core client for 30 sec - exiting 19:20:52 (13433): No heartbeat from core client for 30 sec - exiting 19:20:53 (13433): No heartbeat from core client for 30 sec - exiting 19:20:54 (13433): No heartbeat from core client for 30 sec - exiting 19:20:55 (13433): No heartbeat from core client for 30 sec - exiting 19:20:56 (13433): No heartbeat from core client for 30 sec - exiting 19:20:57 (13433): No heartbeat from core client for 30 sec - exiting 19:20:58 (13433): No heartbeat from core client for 30 sec - exiting 19:20:59 (13433): No heartbeat from core client for 30 sec - exiting 19:21:00 (13433): No heartbeat from core client for 30 sec - exiting 19:21:01 (13433): No heartbeat from core client for 30 sec - exiting 19:21:02 (13433): No heartbeat from core client for 30 sec - exiting 19:21:03 (13433): No heartbeat from core client for 30 sec - exiting 19:21:04 (13433): No heartbeat from core client for 30 sec - exiting 19:21:05 (13433): No heartbeat from core client for 30 sec - exiting 19:21:06 (13433): No heartbeat from core client for 30 sec - exiting 19:21:07 (13433): No heartbeat from core client for 30 sec - exiting 19:21:08 (13433): No heartbeat from core client for 30 sec - exiting 19:21:09 (13433): No heartbeat from core client for 30 sec - exiting 19:21:11 (13433): No heartbeat from core client for 30 sec - exiting 19:21:12 (13433): No heartbeat from core client for 30 sec - exiting 19:21:13 (13433): No heartbeat from core client for 30 sec - exiting 19:21:14 (13433): No heartbeat from core client for 30 sec - exiting 19:21:15 (13433): No heartbeat from core client for 30 sec - exiting 19:21:16 (13433): No heartbeat from core client for 30 sec - exiting 19:21:17 (13433): No heartbeat from core client for 30 sec - exiting 19:21:18 (13433): No heartbeat from core client for 30 sec - exiting 19:21:19 (13433): No heartbeat from core client for 30 sec - exiting 19:21:20 (13433): No heartbeat from core client for 30 sec - exiting 19:21:21 (13433): No heartbeat from core client for 30 sec - exiting 19:21:22 (13433): No heartbeat from core client for 30 sec - exiting 19:21:23 (13433): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C69 Wed Sep 28 19:21:35 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcce8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92458000 - 0x9245ffff /usr/lib/system/libsystem_notify.dylib 0x92b24000 - 0x92b28fff /usr/lib/system/libsystem_network.dylib 0x92bf8000 - 0x92c06fff /usr/lib/libz.1.dylib 0x93278000 - 0x93279fff /usr/lib/system/libunc.dylib 0x9335d000 - 0x9335ffff /usr/lib/system/libdyld.dylib 0x93360000 - 0x93365fff /usr/lib/system/libmacho.dylib 0x9353a000 - 0x9353dfff /usr/lib/system/libcompiler_rt.dylib 0x93571000 - 0x93572fff /usr/lib/system/libremovefile.dylib 0x93aa4000 - 0x93aacfff /usr/lib/system/libcopyfile.dylib 0x93dc8000 - 0x93dc8fff /usr/lib/system/libdnsinfo.dylib 0x9473f000 - 0x9473ffff /usr/lib/system/libkeymgr.dylib 0x94740000 - 0x9475efff /usr/lib/system/libsystem_kernel.dylib 0x94c9e000 - 0x94cacfff /usr/lib/system/libdispatch.dylib 0x94ee6000 - 0x94eeafff /usr/lib/system/libcache.dylib 0x972fc000 - 0x97304fff /usr/lib/system/libunwind.dylib 0x988f7000 - 0x988fafff /usr/lib/system/libmathCommon.A.dylib 0x98b2a000 - 0x98bf5fff /usr/lib/system/libsystem_c.dylib 0x98bf9000 - 0x98c01fff /usr/lib/system/liblaunch.dylib 0x98ec6000 - 0x98ec7fff /usr/lib/system/libquarantine.dylib 0x98ec8000 - 0x98edefff /usr/lib/system/libxpc.dylib 0x98f45000 - 0x98f4cfff /usr/lib/system/libsystem_dnssd.dylib 0x98f82000 - 0x98fb1fff /usr/lib/system/libsystem_info.dylib 0x9a20d000 - 0x9a20efff /usr/lib/system/libsystem_sandbox.dylib 0x9a427000 - 0x9a428fff /usr/lib/system/libsystem_blocks.dylib 0x9a4e6000 - 0x9a529fff /usr/lib/system/libcommonCrypto.dylib 0x9a53b000 - 0x9a569fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13543, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C69 Wed Sep 28 19:21:38 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcce8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92458000 - 0x9245ffff /usr/lib/system/libsystem_notify.dylib 0x92b24000 - 0x92b28fff /usr/lib/system/libsystem_network.dylib 0x92bf8000 - 0x92c06fff /usr/lib/libz.1.dylib 0x93278000 - 0x93279fff /usr/lib/system/libunc.dylib 0x9335d000 - 0x9335ffff /usr/lib/system/libdyld.dylib 0x93360000 - 0x93365fff /usr/lib/system/libmacho.dylib 0x9353a000 - 0x9353dfff /usr/lib/system/libcompiler_rt.dylib 0x93571000 - 0x93572fff /usr/lib/system/libremovefile.dylib 0x93aa4000 - 0x93aacfff /usr/lib/system/libcopyfile.dylib 0x93dc8000 - 0x93dc8fff /usr/lib/system/libdnsinfo.dylib 0x9473f000 - 0x9473ffff /usr/lib/system/libkeymgr.dylib 0x94740000 - 0x9475efff /usr/lib/system/libsystem_kernel.dylib 0x94c9e000 - 0x94cacfff /usr/lib/system/libdispatch.dylib 0x94ee6000 - 0x94eeafff /usr/lib/system/libcache.dylib 0x972fc000 - 0x97304fff /usr/lib/system/libunwind.dylib 0x988f7000 - 0x988fafff /usr/lib/system/libmathCommon.A.dylib 0x98b2a000 - 0x98bf5fff /usr/lib/system/libsystem_c.dylib 0x98bf9000 - 0x98c01fff /usr/lib/system/liblaunch.dylib 0x98ec6000 - 0x98ec7fff /usr/lib/system/libquarantine.dylib 0x98ec8000 - 0x98edefff /usr/lib/system/libxpc.dylib 0x98f45000 - 0x98f4cfff /usr/lib/system/libsystem_dnssd.dylib 0x98f82000 - 0x98fb1fff /usr/lib/system/libsystem_info.dylib 0x9a20d000 - 0x9a20efff /usr/lib/system/libsystem_sandbox.dylib 0x9a427000 - 0x9a428fff /usr/lib/system/libsystem_blocks.dylib 0x9a4e6000 - 0x9a529fff /usr/lib/system/libcommonCrypto.dylib 0x9a53b000 - 0x9a569fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13543, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C69 Wed Sep 28 19:21:41 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcce8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92458000 - 0x9245ffff /usr/lib/system/libsystem_notify.dylib 0x92b24000 - 0x92b28fff /usr/lib/system/libsystem_network.dylib 0x92bf8000 - 0x92c06fff /usr/lib/libz.1.dylib 0x93278000 - 0x93279fff /usr/lib/system/libunc.dylib 0x9335d000 - 0x9335ffff /usr/lib/system/libdyld.dylib 0x93360000 - 0x93365fff /usr/lib/system/libmacho.dylib 0x9353a000 - 0x9353dfff /usr/lib/system/libcompiler_rt.dylib 0x93571000 - 0x93572fff /usr/lib/system/libremovefile.dylib 0x93aa4000 - 0x93aacfff /usr/lib/system/libcopyfile.dylib 0x93dc8000 - 0x93dc8fff /usr/lib/system/libdnsinfo.dylib 0x9473f000 - 0x9473ffff /usr/lib/system/libkeymgr.dylib 0x94740000 - 0x9475efff /usr/lib/system/libsystem_kernel.dylib 0x94c9e000 - 0x94cacfff /usr/lib/system/libdispatch.dylib 0x94ee6000 - 0x94eeafff /usr/lib/system/libcache.dylib 0x972fc000 - 0x97304fff /usr/lib/system/libunwind.dylib 0x988f7000 - 0x988fafff /usr/lib/system/libmathCommon.A.dylib 0x98b2a000 - 0x98bf5fff /usr/lib/system/libsystem_c.dylib 0x98bf9000 - 0x98c01fff /usr/lib/system/liblaunch.dylib 0x98ec6000 - 0x98ec7fff /usr/lib/system/libquarantine.dylib 0x98ec8000 - 0x98edefff /usr/lib/system/libxpc.dylib 0x98f45000 - 0x98f4cfff /usr/lib/system/libsystem_dnssd.dylib 0x98f82000 - 0x98fb1fff /usr/lib/system/libsystem_info.dylib 0x9a20d000 - 0x9a20efff /usr/lib/system/libsystem_sandbox.dylib 0x9a427000 - 0x9a428fff /usr/lib/system/libsystem_blocks.dylib 0x9a4e6000 - 0x9a529fff /usr/lib/system/libcommonCrypto.dylib 0x9a53b000 - 0x9a569fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13543, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C69 Wed Sep 28 19:21:44 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcce8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92458000 - 0x9245ffff /usr/lib/system/libsystem_notify.dylib 0x92b24000 - 0x92b28fff /usr/lib/system/libsystem_network.dylib 0x92bf8000 - 0x92c06fff /usr/lib/libz.1.dylib 0x93278000 - 0x93279fff /usr/lib/system/libunc.dylib 0x9335d000 - 0x9335ffff /usr/lib/system/libdyld.dylib 0x93360000 - 0x93365fff /usr/lib/system/libmacho.dylib 0x9353a000 - 0x9353dfff /usr/lib/system/libcompiler_rt.dylib 0x93571000 - 0x93572fff /usr/lib/system/libremovefile.dylib 0x93aa4000 - 0x93aacfff /usr/lib/system/libcopyfile.dylib 0x93dc8000 - 0x93dc8fff /usr/lib/system/libdnsinfo.dylib 0x9473f000 - 0x9473ffff /usr/lib/system/libkeymgr.dylib 0x94740000 - 0x9475efff /usr/lib/system/libsystem_kernel.dylib 0x94c9e000 - 0x94cacfff /usr/lib/system/libdispatch.dylib 0x94ee6000 - 0x94eeafff /usr/lib/system/libcache.dylib 0x972fc000 - 0x97304fff /usr/lib/system/libunwind.dylib 0x988f7000 - 0x988fafff /usr/lib/system/libmathCommon.A.dylib 0x98b2a000 - 0x98bf5fff /usr/lib/system/libsystem_c.dylib 0x98bf9000 - 0x98c01fff /usr/lib/system/liblaunch.dylib 0x98ec6000 - 0x98ec7fff /usr/lib/system/libquarantine.dylib 0x98ec8000 - 0x98edefff /usr/lib/system/libxpc.dylib 0x98f45000 - 0x98f4cfff /usr/lib/system/libsystem_dnssd.dylib 0x98f82000 - 0x98fb1fff /usr/lib/system/libsystem_info.dylib 0x9a20d000 - 0x9a20efff /usr/lib/system/libsystem_sandbox.dylib 0x9a427000 - 0x9a428fff /usr/lib/system/libsystem_blocks.dylib 0x9a4e6000 - 0x9a529fff /usr/lib/system/libcommonCrypto.dylib 0x9a53b000 - 0x9a569fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13543, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C69 Wed Sep 28 19:21:47 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcce8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92458000 - 0x9245ffff /usr/lib/system/libsystem_notify.dylib 0x92b24000 - 0x92b28fff /usr/lib/system/libsystem_network.dylib 0x92bf8000 - 0x92c06fff /usr/lib/libz.1.dylib 0x93278000 - 0x93279fff /usr/lib/system/libunc.dylib 0x9335d000 - 0x9335ffff /usr/lib/system/libdyld.dylib 0x93360000 - 0x93365fff /usr/lib/system/libmacho.dylib 0x9353a000 - 0x9353dfff /usr/lib/system/libcompiler_rt.dylib 0x93571000 - 0x93572fff /usr/lib/system/libremovefile.dylib 0x93aa4000 - 0x93aacfff /usr/lib/system/libcopyfile.dylib 0x93dc8000 - 0x93dc8fff /usr/lib/system/libdnsinfo.dylib 0x9473f000 - 0x9473ffff /usr/lib/system/libkeymgr.dylib 0x94740000 - 0x9475efff /usr/lib/system/libsystem_kernel.dylib 0x94c9e000 - 0x94cacfff /usr/lib/system/libdispatch.dylib 0x94ee6000 - 0x94eeafff /usr/lib/system/libcache.dylib 0x972fc000 - 0x97304fff /usr/lib/system/libunwind.dylib 0x988f7000 - 0x988fafff /usr/lib/system/libmathCommon.A.dylib 0x98b2a000 - 0x98bf5fff /usr/lib/system/libsystem_c.dylib 0x98bf9000 - 0x98c01fff /usr/lib/system/liblaunch.dylib 0x98ec6000 - 0x98ec7fff /usr/lib/system/libquarantine.dylib 0x98ec8000 - 0x98edefff /usr/lib/system/libxpc.dylib 0x98f45000 - 0x98f4cfff /usr/lib/system/libsystem_dnssd.dylib 0x98f82000 - 0x98fb1fff /usr/lib/system/libsystem_info.dylib 0x9a20d000 - 0x9a20efff /usr/lib/system/libsystem_sandbox.dylib 0x9a427000 - 0x9a428fff /usr/lib/system/libsystem_blocks.dylib 0x9a4e6000 - 0x9a529fff /usr/lib/system/libcommonCrypto.dylib 0x9a53b000 - 0x9a569fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13543, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C69 Wed Sep 28 19:21:51 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcce8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92458000 - 0x9245ffff /usr/lib/system/libsystem_notify.dylib 0x92b24000 - 0x92b28fff /usr/lib/system/libsystem_network.dylib 0x92bf8000 - 0x92c06fff /usr/lib/libz.1.dylib 0x93278000 - 0x93279fff /usr/lib/system/libunc.dylib 0x9335d000 - 0x9335ffff /usr/lib/system/libdyld.dylib 0x93360000 - 0x93365fff /usr/lib/system/libmacho.dylib 0x9353a000 - 0x9353dfff /usr/lib/system/libcompiler_rt.dylib 0x93571000 - 0x93572fff /usr/lib/system/libremovefile.dylib 0x93aa4000 - 0x93aacfff /usr/lib/system/libcopyfile.dylib 0x93dc8000 - 0x93dc8fff /usr/lib/system/libdnsinfo.dylib 0x9473f000 - 0x9473ffff /usr/lib/system/libkeymgr.dylib 0x94740000 - 0x9475efff /usr/lib/system/libsystem_kernel.dylib 0x94c9e000 - 0x94cacfff /usr/lib/system/libdispatch.dylib 0x94ee6000 - 0x94eeafff /usr/lib/system/libcache.dylib 0x972fc000 - 0x97304fff /usr/lib/system/libunwind.dylib 0x988f7000 - 0x988fafff /usr/lib/system/libmathCommon.A.dylib 0x98b2a000 - 0x98bf5fff /usr/lib/system/libsystem_c.dylib 0x98bf9000 - 0x98c01fff /usr/lib/system/liblaunch.dylib 0x98ec6000 - 0x98ec7fff /usr/lib/system/libquarantine.dylib 0x98ec8000 - 0x98edefff /usr/lib/system/libxpc.dylib 0x98f45000 - 0x98f4cfff /usr/lib/system/libsystem_dnssd.dylib 0x98f82000 - 0x98fb1fff /usr/lib/system/libsystem_info.dylib 0x9a20d000 - 0x9a20efff /usr/lib/system/libsystem_sandbox.dylib 0x9a427000 - 0x9a428fff /usr/lib/system/libsystem_blocks.dylib 0x9a4e6000 - 0x9a529fff /usr/lib/system/libcommonCrypto.dylib 0x9a53b000 - 0x9a569fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13543, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Sep 2011 17:48:52	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	362,880	735,650	2.0273
27 Sep 2011 20:44:57	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	336,960	684,310	2.0308
27 Sep 2011 05:05:37	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	311,040	632,303	2.0329
26 Sep 2011 11:21:23	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	285,120	580,254	2.0351
25 Sep 2011 15:55:02	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	259,200	527,672	2.0358
24 Sep 2011 22:22:04	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	233,280	474,514	2.0341
24 Sep 2011 02:37:33	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	207,360	421,916	2.0347
23 Sep 2011 01:45:10	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	181,440	369,403	2.0360
22 Sep 2011 07:01:33	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	155,520	316,079	2.0324
21 Sep 2011 07:05:22	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	129,600	263,417	2.0325
20 Sep 2011 11:40:05	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	103,680	211,081	2.0359
19 Sep 2011 15:37:52	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	77,760	158,271	2.0354
18 Sep 2011 17:33:04	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	51,840	105,303	2.0313
17 Sep 2011 18:24:40	615938	13358146	hadcm3n_o55h_1940_40_007446504_2	25,920	52,538	2.0269