Task 13349510

Name	hadcm3n_o4xa_1940_40_007443671_1
Workunit	7641174
Created	9 Sep 2011, 0:28:15 UTC
Sent	9 Sep 2011, 0:34:33 UTC
Report deadline	9 Dec 2011, 8:01:44 UTC
Received	27 Sep 2011, 19:27:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1168499
Run time	13 days 12 hours 49 min 46 sec
CPU time	11 days 18 hours 36 min 22 sec
Validate state	Invalid
Credit	6,531.84
Device peak FLOPS	3.28 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(284,0xac3252c0) malloc: * error for object 0xb23604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Mon Sep 26 17:25:42 2011 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=298, selfPID=298, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x3a26604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x3a26600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41f9600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x41faa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x4209004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x4217604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30725,0xac3252c0) malloc: * error for object 0x4217600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:14:58 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38641, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:15:01 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:16:33 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38680, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:16:37 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38680, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:16:40 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:21:11 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38745, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:21:15 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38745, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.1 build 11B26 Tue Sep 27 12:21:18 2011 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x90671000 - 0x90678fff /usr/lib/system/libsystem_dnssd.dylib 0x91a23000 - 0x91a23fff /usr/lib/system/libdnsinfo.dylib 0x92042000 - 0x92043fff /usr/lib/system/libremovefile.dylib 0x923ba000 - 0x923bdfff /usr/lib/system/libmathCommon.A.dylib 0x92925000 - 0x929f0fff /usr/lib/system/libsystem_c.dylib 0x92ef4000 - 0x92ef4fff /usr/lib/system/libkeymgr.dylib 0x92ef5000 - 0x92f25fff /usr/lib/system/libsystem_info.dylib 0x93c36000 - 0x93c37fff /usr/lib/system/libsystem_blocks.dylib 0x94302000 - 0x9430afff /usr/lib/system/liblaunch.dylib 0x94ddf000 - 0x94dedfff /usr/lib/libz.1.dylib 0x956d7000 - 0x956dbfff /usr/lib/system/libsystem_network.dylib 0x95fde000 - 0x95fe6fff /usr/lib/system/libcopyfile.dylib 0x96062000 - 0x96070fff /usr/lib/system/libdispatch.dylib 0x96a3c000 - 0x96a43fff /usr/lib/system/libnotify.dylib 0x96f0d000 - 0x96f0efff /usr/lib/system/libunc.dylib 0x977e1000 - 0x977e3fff /usr/lib/system/libdyld.dylib 0x97932000 - 0x97936fff /usr/lib/system/libcache.dylib 0x9803c000 - 0x9805afff /usr/lib/system/libsystem_kernel.dylib 0x98cdc000 - 0x98d0afff /usr/lib/libSystem.B.dylib 0x98da4000 - 0x98dbafff /usr/lib/system/libxpc.dylib 0x9a6eb000 - 0x9a6ecfff /usr/lib/system/libsystem_sandbox.dylib 0x9a741000 - 0x9a784fff /usr/lib/system/libcommonCrypto.dylib 0x9b4ed000 - 0x9b4f0fff /usr/lib/system/libcompiler_rt.dylib 0x9b63d000 - 0x9b642fff /usr/lib/system/libmacho.dylib 0x9b916000 - 0x9b91efff /usr/lib/system/libunwind.dylib 0x9bbcd000 - 0x9bbcefff /usr/lib/system/libquarantine.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38745, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Sep 2011 17:32:33	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	544,320	1,014,100	1.8631
27 Sep 2011 00:42:40	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	518,400	965,661	1.8628
26 Sep 2011 08:59:54	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	492,480	917,615	1.8633
25 Sep 2011 16:30:33	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	466,560	869,290	1.8632
25 Sep 2011 00:40:15	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	440,640	821,843	1.8651
24 Sep 2011 09:06:38	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	414,720	774,331	1.8671
23 Sep 2011 11:42:07	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	388,800	725,596	1.8662
22 Sep 2011 13:11:37	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	362,880	676,660	1.8647
20 Sep 2011 21:29:05	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	336,960	628,131	1.8641
20 Sep 2011 00:21:31	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	311,040	578,398	1.8596
19 Sep 2011 19:47:18	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	285,120	529,929	1.8586
19 Sep 2011 19:47:18	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	259,200	480,809	1.8550
19 Sep 2011 19:47:18	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	233,280	431,100	1.8480
17 Sep 2011 04:38:22	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	207,360	382,837	1.8462
14 Sep 2011 14:06:48	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	181,440	334,388	1.8430
13 Sep 2011 23:32:47	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	155,520	286,479	1.8421
13 Sep 2011 06:06:51	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	129,600	239,566	1.8485
12 Sep 2011 10:42:58	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	103,680	192,971	1.8612
11 Sep 2011 19:14:57	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	77,760	143,667	1.8476
11 Sep 2011 04:36:24	1168499	13349510	hadcm3n_o4xa_1940_40_007443671_1	51,840	95,703	1.8461