Task 13581105

Name	hadcm3n_y9sy_1900_40_007524907_4
Workunit	7722382
Created	1 Nov 2011, 12:14:50 UTC
Sent	1 Nov 2011, 12:19:54 UTC
Report deadline	31 Jan 2012, 19:47:05 UTC
Received	20 Dec 2011, 14:27:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1032461
Run time	13 days 2 hours 49 min 49 sec
CPU time	10 days 15 hours 43 min 37 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	3.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:30:20 (833): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 11:34:01 (5773): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:23:30 (3337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:13:01 (25665): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:40:12 (279): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:41:03 (836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:41 (845): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:31:36 (856): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:29:18 (1395): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:38:51 (22603): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:26 (32640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:18 (32764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:45:02 (32774): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:34:04 (288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:14 (939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:24:05 (950): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:05 (21790): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:10:06 (21805): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 10:11:22 (50449): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:39 (50563): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:40 (50563): No heartbeat from core client for 30 sec - exiting 15:02:41 (50563): No heartbeat from core client for 30 sec - exiting 15:02:42 (50563): No heartbeat from core client for 30 sec - exiting 15:02:43 (50563): No heartbeat from core client for 30 sec - exiting 15:02:44 (50563): No heartbeat from core client for 30 sec - exiting 15:02:45 (50563): No heartbeat from core client for 30 sec - exiting 15:02:46 (50563): No heartbeat from core client for 30 sec - exiting 15:02:47 (50563): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Dec 19 04:42:30 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001437c5 _impl_cal + 9669 1 hadcm3n_um_6.07_i686-apple-darwin 0x0004b675 _bdy_layr + 8149 2 hadcm3n_um_6.07_i686-apple-darwin 0x00073922 _bl_intct + 1394 3 hadcm3n_um_6.07_i686-apple-darwin 0x000647fc _bl_ctl + 5356 4 hadcm3n_um_6.07_i686-apple-darwin 0x000415a2 _atm_phys + 11234 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9a7ffc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a887429 _nanosleep + 345 2 libSystem.B.dylib 0x9a8872ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a82d259 __pthread_start + 345 6 libSystem.B.dylib 0x9a82d0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe7a3148 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x94b97000 - 0x94b9afff /usr/lib/system/libmathCommon.A.dylib 0x960e2000 - 0x960f0fff /usr/lib/libz.1.dylib 0x9a7ff000 - 0x9a9a6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50581, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Dec 19 04:42:36 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001437c5 _impl_cal + 9669 1 hadcm3n_um_6.07_i686-apple-darwin 0x0004b675 _bdy_layr + 8149 2 hadcm3n_um_6.07_i686-apple-darwin 0x00073922 _bl_intct + 1394 3 hadcm3n_um_6.07_i686-apple-darwin 0x000647fc _bl_ctl + 5356 4 hadcm3n_um_6.07_i686-apple-darwin 0x000415a2 _atm_phys + 11234 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9a7ffc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a887429 _nanosleep + 345 2 libSystem.B.dylib 0x9a8872ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a82d259 __pthread_start + 345 6 libSystem.B.dylib 0x9a82d0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe7a3148 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x94b97000 - 0x94b9afff /usr/lib/system/libmathCommon.A.dylib 0x960e2000 - 0x960f0fff /usr/lib/libz.1.dylib 0x9a7ff000 - 0x9a9a6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50581, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Dec 19 04:42:43 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001437c5 _impl_cal + 9669 1 hadcm3n_um_6.07_i686-apple-darwin 0x0004b675 _bdy_layr + 8149 2 hadcm3n_um_6.07_i686-apple-darwin 0x00073922 _bl_intct + 1394 3 hadcm3n_um_6.07_i686-apple-darwin 0x000647fc _bl_ctl + 5356 4 hadcm3n_um_6.07_i686-apple-darwin 0x000415a2 _atm_phys + 11234 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9a7ffc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a887429 _nanosleep + 345 2 libSystem.B.dylib 0x9a8872ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a82d259 __pthread_start + 345 6 libSystem.B.dylib 0x9a82d0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe7a3148 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x94b97000 - 0x94b9afff /usr/lib/system/libmathCommon.A.dylib 0x960e2000 - 0x960f0fff /usr/lib/libz.1.dylib 0x9a7ff000 - 0x9a9a6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50581, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Dec 19 04:42:49 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001437c5 _impl_cal + 9669 1 hadcm3n_um_6.07_i686-apple-darwin 0x0004b675 _bdy_layr + 8149 2 hadcm3n_um_6.07_i686-apple-darwin 0x00073922 _bl_intct + 1394 3 hadcm3n_um_6.07_i686-apple-darwin 0x000647fc _bl_ctl + 5356 4 hadcm3n_um_6.07_i686-apple-darwin 0x000415a2 _atm_phys + 11234 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9a7ffc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a887429 _nanosleep + 345 2 libSystem.B.dylib 0x9a8872ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a82d259 __pthread_start + 345 6 libSystem.B.dylib 0x9a82d0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe7a3148 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x94b97000 - 0x94b9afff /usr/lib/system/libmathCommon.A.dylib 0x960e2000 - 0x960f0fff /usr/lib/libz.1.dylib 0x9a7ff000 - 0x9a9a6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50581, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Dec 19 04:42:55 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001437c5 _impl_cal + 9669 1 hadcm3n_um_6.07_i686-apple-darwin 0x0004b675 _bdy_layr + 8149 2 hadcm3n_um_6.07_i686-apple-darwin 0x00073922 _bl_intct + 1394 3 hadcm3n_um_6.07_i686-apple-darwin 0x000647fc _bl_ctl + 5356 4 hadcm3n_um_6.07_i686-apple-darwin 0x000415a2 _atm_phys + 11234 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9a7ffc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a887429 _nanosleep + 345 2 libSystem.B.dylib 0x9a8872ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a82d259 __pthread_start + 345 6 libSystem.B.dylib 0x9a82d0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe7a3148 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x94b97000 - 0x94b9afff /usr/lib/system/libmathCommon.A.dylib 0x960e2000 - 0x960f0fff /usr/lib/libz.1.dylib 0x9a7ff000 - 0x9a9a6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50581, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Dec 19 04:43:01 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x001437c5 _impl_cal + 9669 1 hadcm3n_um_6.07_i686-apple-darwin 0x0004b675 _bdy_layr + 8149 2 hadcm3n_um_6.07_i686-apple-darwin 0x00073922 _bl_intct + 1394 3 hadcm3n_um_6.07_i686-apple-darwin 0x000647fc _bl_ctl + 5356 4 hadcm3n_um_6.07_i686-apple-darwin 0x000415a2 _atm_phys + 11234 5 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 6 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 7 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 8 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 9 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 10 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x9a7ffc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a887429 _nanosleep + 345 2 libSystem.B.dylib 0x9a8872ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a82d259 __pthread_start + 345 6 libSystem.B.dylib 0x9a82d0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe7a3148 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x94b97000 - 0x94b9afff /usr/lib/system/libmathCommon.A.dylib 0x960e2000 - 0x960f0fff /usr/lib/libz.1.dylib 0x9a7ff000 - 0x9a9a6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50581, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2011 02:26:15	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	803,520	977,572	1.2166
18 Dec 2011 08:57:32	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	777,600	947,512	1.2185
16 Dec 2011 23:11:18	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	751,680	917,436	1.2205
16 Dec 2011 11:22:16	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	725,760	887,775	1.2232
12 Dec 2011 05:02:25	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	699,840	856,853	1.2244
08 Dec 2011 14:14:34	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	673,920	824,648	1.2237
06 Dec 2011 09:25:10	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	648,000	791,673	1.2217
04 Dec 2011 18:07:29	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	622,080	761,102	1.2235
04 Dec 2011 00:22:58	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	596,160	728,353	1.2217
04 Dec 2011 00:22:58	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	570,240	694,385	1.2177
01 Dec 2011 06:24:58	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	544,320	660,180	1.2129
30 Nov 2011 13:58:04	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	518,400	629,592	1.2145
30 Nov 2011 13:58:04	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	492,480	599,514	1.2173
28 Nov 2011 02:18:38	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	466,560	567,435	1.2162
26 Nov 2011 23:32:56	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	440,640	535,666	1.2157
25 Nov 2011 06:20:18	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	414,720	505,106	1.2179
24 Nov 2011 13:32:57	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	388,800	474,989	1.2217
24 Nov 2011 04:40:30	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	362,880	444,861	1.2259
22 Nov 2011 03:37:35	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	336,960	414,229	1.2293
22 Nov 2011 03:37:35	1032461	13581105	hadcm3n_y9sy_1900_40_007524907_4	311,040	384,048	1.2347