Task 15817239

Name	hadcm3n_zhl4_1920_40_008315744_3
Workunit	8466879
Created	2 Jun 2013, 2:05:31 UTC
Sent	2 Jun 2013, 2:20:39 UTC
Report deadline	1 Sep 2013, 9:47:50 UTC
Received	3 Jul 2013, 9:34:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1231328
Run time	17 days 3 hours 23 min 13 sec
CPU time	10 days 3 hours 0 min 46 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.02 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 04:49:06 (5635): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:13 (5635): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:05:51 (15106): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:10:01 (25950): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:03 (25950): No heartbeat from core client for 30 sec - exiting 05:26:46 (30436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:05 (30570): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:10 (30570): No heartbeat from core client for 30 sec - exiting 06:00:11 (30570): No heartbeat from core client for 30 sec - exiting 06:00:12 (30570): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:41:42 (45953): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:01:33 (3319): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:01:35 (3319): No heartbeat from core client for 30 sec - exiting 19:01:36 (3319): No heartbeat from core client for 30 sec - exiting 19:01:37 (3319): No heartbeat from core client for 30 sec - exiting 19:01:38 (3319): No heartbeat from core client for 30 sec - exiting 19:01:39 (3319): No heartbeat from core client for 30 sec - exiting 19:01:40 (3319): No heartbeat from core client for 30 sec - exiting 19:01:41 (3319): No heartbeat from core client for 30 sec - exiting 19:01:42 (3319): No heartbeat from core client for 30 sec - exiting 19:01:43 (3319): No heartbeat from core client for 30 sec - exiting 19:01:44 (3319): No heartbeat from core client for 30 sec - exiting 19:01:45 (3319): No heartbeat from core client for 30 sec - exiting 19:01:46 (3319): No heartbeat from core client for 30 sec - exiting 19:01:47 (3319): No heartbeat from core client for 30 sec - exiting 19:01:48 (3319): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:48:18 (59061): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x1036c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x1036c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x3024604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x3024600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x303fc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x303fc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(41797,0xa05eb540) malloc: * error for object 0x827004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Jun 17 22:28:10 2013 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 0 libSystem.B.dylib 0x94023ac5 small_malloc_from_free_list + 133 1 libSystem.B.dylib 0x940204d9 szone_malloc_should_clear + 2271 2 libSystem.B.dylib 0x9401fba8 malloc_zone_malloc + 81 3 libSystem.B.dylib 0x9401dc78 malloc + 50 4 libstdc++.6.dylib 0x90639617 operator new(unsigned long) + 36 5 libstdc++.6.dylib 0x90639703 operator new[](unsigned long) + 17 6 hadcm3n_6.07_i686-apple-darwin 0x0000b8e7 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3113 7 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 8 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 9 hadcm3n_6.07_i686-apple-darwin 0x0000791c mainLoop() + 410 10 hadcm3n_6.07_i686-apple-darwin 0x000087c7 worker() + 2989 11 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 12 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: error for object 0x2805c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x2805c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x2805c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x2805c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x2805c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x2805c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x5005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x5005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x5005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x5005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x5005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(51870,0xa05eb540) malloc: * error for object 0x5005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 00:08:42 (62800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:08:45 (62800): No heartbeat from core client for 30 sec - exiting 00:08:46 (62800): No heartbeat from core client for 30 sec - exiting 00:08:47 (62800): No heartbeat from core client for 30 sec - exiting 00:08:48 (62800): No heartbeat from core client for 30 sec - exiting 00:08:49 (62800): No heartbeat from core client for 30 sec - exiting 00:08:50 (62800): No heartbeat from core client for 30 sec - exiting 00:08:51 (62800): No heartbeat from core client for 30 sec - exiting 00:08:52 (62800): No heartbeat from core client for 30 sec - exiting 00:08:53 (62800): No heartbeat from core client for 30 sec - exiting 00:08:54 (62800): No heartbeat from core client for 30 sec - exiting 00:08:55 (62800): No heartbeat from core client for 30 sec - exiting 00:08:56 (62800): No heartbeat from core client for 30 sec - exiting 00:08:57 (62800): No heartbeat from core client for 30 sec - exiting 00:08:58 (62800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:44:04 (3186): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:44:05 (3186): No heartbeat from core client for 30 sec - exiting 22:44:06 (3186): No heartbeat from core client for 30 sec - exiting 22:44:07 (3186): No heartbeat from core client for 30 sec - exiting 22:44:08 (3186): No heartbeat from core client for 30 sec - exiting 22:44:09 (3186): No heartbeat from core client for 30 sec - exiting 22:44:10 (3186): No heartbeat from core client for 30 sec - exiting 22:44:11 (3186): No heartbeat from core client for 30 sec - exiting 22:44:12 (3186): No heartbeat from core client for 30 sec - exiting 22:44:13 (3186): No heartbeat from core client for 30 sec - exiting 22:44:14 (3186): No heartbeat from core client for 30 sec - exiting 22:44:15 (3186): No heartbeat from core client for 30 sec - exiting 22:44:16 (3186): No heartbeat from core client for 30 sec - exiting 22:44:17 (3186): No heartbeat from core client for 30 sec - exiting 22:44:18 (3186): No heartbeat from core client for 30 sec - exiting 22:44:19 (3186): No heartbeat from core client for 30 sec - exiting 22:44:20 (3186): No heartbeat from core client for 30 sec - exiting 22:44:21 (3186): No heartbeat from core client for 30 sec - exiting 22:44:22 (3186): No heartbeat from core client for 30 sec - exiting 22:44:23 (3186): No heartbeat from core client for 30 sec - exiting 22:44:24 (3186): No heartbeat from core client for 30 sec - exiting 22:44:25 (3186): No heartbeat from core client for 30 sec - exiting 23:45:39 (5151): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:36:21 (13274): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:43 (14780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:44 (14780): No heartbeat from core client for 30 sec - exiting 21:38:45 (14780): No heartbeat from core client for 30 sec - exiting 21:38:46 (14780): No heartbeat from core client for 30 sec - exiting 21:38:47 (14780): No heartbeat from core client for 30 sec - exiting 21:38:48 (14780): No heartbeat from core client for 30 sec - exiting 21:38:49 (14780): No heartbeat from core client for 30 sec - exiting 21:38:50 (14780): No heartbeat from core client for 30 sec - exiting 21:38:51 (14780): No heartbeat from core client for 30 sec - exiting 21:38:52 (14780): No heartbeat from core client for 30 sec - exiting 21:38:53 (14780): No heartbeat from core client for 30 sec - exiting 21:38:54 (14780): No heartbeat from core client for 30 sec - exiting 21:38:55 (14780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:58:54 (23509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:56 (24333): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:57 (24333): No heartbeat from core client for 30 sec - exiting 01:35:58 (24333): No heartbeat from core client for 30 sec - exiting 01:35:59 (24333): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:30:36 (44747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:30:37 (44747): No heartbeat from core client for 30 sec - exiting 04:30:38 (44747): No heartbeat from core client for 30 sec - exiting 04:30:39 (44747): No heartbeat from core client for 30 sec - exiting 04:30:40 (44747): No heartbeat from core client for 30 sec - exiting 04:30:41 (44747): No heartbeat from core client for 30 sec - exiting 04:30:42 (44747): No heartbeat from core client for 30 sec - exiting 04:30:43 (44747): No heartbeat from core client for 30 sec - exiting 04:30:44 (44747): No heartbeat from core client for 30 sec - exiting 04:30:45 (44747): No heartbeat from core client for 30 sec - exiting 04:30:46 (44747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:12:19 (54786): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:20 (54786): No heartbeat from core client for 30 sec - exiting 19:12:21 (54786): No heartbeat from core client for 30 sec - exiting 04:16:02 (55302): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:16:03 (55302): No heartbeat from core client for 30 sec - exiting 04:16:04 (55302): No heartbeat from core client for 30 sec - exiting 04:16:05 (55302): No heartbeat from core client for 30 sec - exiting 04:16:06 (55302): No heartbeat from core client for 30 sec - exiting 04:16:08 (55302): No heartbeat from core client for 30 sec - exiting 04:16:09 (55302): No heartbeat from core client for 30 sec - exiting 04:16:10 (55302): No heartbeat from core client for 30 sec - exiting 04:16:11 (55302): No heartbeat from core client for 30 sec - exiting 04:16:12 (55302): No heartbeat from core client for 30 sec - exiting 04:16:13 (55302): No heartbeat from core client for 30 sec - exiting 04:16:14 (55302): No heartbeat from core client for 30 sec - exiting 04:16:15 (55302): No heartbeat from core client for 30 sec - exiting 04:16:16 (55302): No heartbeat from core client for 30 sec - exiting 04:16:17 (55302): No heartbeat from core client for 30 sec - exiting 04:16:18 (55302): No heartbeat from core client for 30 sec - exiting 04:16:19 (55302): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:27:31 (85820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:33 (85820): No heartbeat from core client for 30 sec - exiting 23:27:34 (85820): No heartbeat from core client for 30 sec - exiting 23:27:35 (85820): No heartbeat from core client for 30 sec - exiting 23:27:36 (85820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:03:11 (96545): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Tue Jul 2 18:03:21 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9407cc5a __kill + 10 1 libSystem.B.dylib 0x9410f5a5 raise + 26 2 libSystem.B.dylib 0x941256e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9401bc0e mach_wait_until + 10 1 libSystem.B.dylib 0x940a3429 nanosleep + 345 2 libSystem.B.dylib 0x940a32ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void) + 78 5 libSystem.B.dylib 0x94049259 _pthread_start + 345 6 libSystem.B.dylib 0x940490de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9401b000 - 0x941c2fff /usr/lib/libSystem.B.dylib 0x97c98000 - 0x97ca6fff /usr/lib/libz.1.dylib 0x99fcd000 - 0x99fd0fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=96590, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Tue Jul 2 18:03:24 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9407cc5a __kill + 10 1 libSystem.B.dylib 0x9410f5a5 raise + 26 2 libSystem.B.dylib 0x941256e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9401bc0e mach_wait_until + 10 1 libSystem.B.dylib 0x940a3429 nanosleep + 345 2 libSystem.B.dylib 0x940a32ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void) + 78 5 libSystem.B.dylib 0x94049259 _pthread_start + 345 6 libSystem.B.dylib 0x940490de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9401b000 - 0x941c2fff /usr/lib/libSystem.B.dylib 0x97c98000 - 0x97ca6fff /usr/lib/libz.1.dylib 0x99fcd000 - 0x99fd0fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=96590, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Tue Jul 2 18:03:27 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9407cc5a __kill + 10 1 libSystem.B.dylib 0x9410f5a5 raise + 26 2 libSystem.B.dylib 0x941256e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9401bc0e mach_wait_until + 10 1 libSystem.B.dylib 0x940a3429 nanosleep + 345 2 libSystem.B.dylib 0x940a32ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void) + 78 5 libSystem.B.dylib 0x94049259 _pthread_start + 345 6 libSystem.B.dylib 0x940490de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9401b000 - 0x941c2fff /usr/lib/libSystem.B.dylib 0x97c98000 - 0x97ca6fff /usr/lib/libz.1.dylib 0x99fcd000 - 0x99fd0fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=96590, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Tue Jul 2 18:03:30 2013 Thread 0 Crashed: 01:39:25 (96590): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Jul 3 01:39:35 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9407cc5a __kill + 10 1 libSystem.B.dylib 0x9410f5a5 raise + 26 2 libSystem.B.dylib 0x941256e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9401bc0e mach_wait_until + 10 1 libSystem.B.dylib 0x940a3429 nanosleep + 345 2 libSystem.B.dylib 0x940a32ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void) + 78 5 libSystem.B.dylib 0x94049259 _pthread_start + 345 6 libSystem.B.dylib 0x940490de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9401b000 - 0x941c2fff /usr/lib/libSystem.B.dylib 0x97c98000 - 0x97ca6fff /usr/lib/libz.1.dylib 0x99fcd000 - 0x99fd0fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=99850, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Jul 3 01:39:39 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9407cc5a __kill + 10 1 libSystem.B.dylib 0x9410f5a5 raise + 26 2 libSystem.B.dylib 0x941256e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9401bc0e mach_wait_until + 10 1 libSystem.B.dylib 0x940a3429 nanosleep + 345 2 libSystem.B.dylib 0x940a32ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void) + 78 5 libSystem.B.dylib 0x94049259 _pthread_start + 345 6 libSystem.B.dylib 0x940490de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9401b000 - 0x941c2fff /usr/lib/libSystem.B.dylib 0x97c98000 - 0x97ca6fff /usr/lib/libz.1.dylib 0x99fcd000 - 0x99fd0fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=99850, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Wed Jul 3 01:39:42 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9407cc5a __kill + 10 1 libSystem.B.dylib 0x9410f5a5 raise + 26 2 libSystem.B.dylib 0x941256e4 abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb _start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9401bc0e mach_wait_until + 10 1 libSystem.B.dylib 0x940a3429 nanosleep + 345 2 libSystem.B.dylib 0x940a32ca usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 boinc_sleep(double) + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e timer_thread(void) + 78 5 libSystem.B.dylib 0x94049259 _pthread_start + 345 6 libSystem.B.dylib 0x940490de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9401b000 - 0x941c2fff /usr/lib/libSystem.B.dylib 0x97c98000 - 0x97ca6fff /usr/lib/libz.1.dylib 0x99fcd000 - 0x99fd0fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=99850, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2013 12:05:35	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	466,560	857,928	1.8388
02 Jul 2013 10:05:30	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	440,640	809,749	1.8377
28 Jun 2013 03:06:00	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	414,720	762,923	1.8396
26 Jun 2013 07:44:04	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	388,800	714,740	1.8383
24 Jun 2013 13:01:51	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	362,880	666,955	1.8379
23 Jun 2013 04:14:47	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	336,960	618,856	1.8366
21 Jun 2013 10:04:00	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	311,040	570,533	1.8343
19 Jun 2013 14:33:57	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	285,120	522,118	1.8312
18 Jun 2013 05:32:17	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	259,200	474,128	1.8292
16 Jun 2013 10:56:43	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	233,280	426,633	1.8288
15 Jun 2013 02:02:00	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	207,360	378,867	1.8271
13 Jun 2013 08:13:36	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	181,440	332,541	1.8328
11 Jun 2013 14:05:31	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	155,520	284,855	1.8316
10 Jun 2013 05:06:22	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	129,600	237,283	1.8309
08 Jun 2013 10:19:17	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	103,680	190,187	1.8344
07 Jun 2013 01:42:27	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	77,760	142,572	1.8335
05 Jun 2013 07:04:37	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	51,840	95,002	1.8326
03 Jun 2013 11:10:36	1231328	15817239	hadcm3n_zhl4_1920_40_008315744_3	25,920	47,636	1.8378