Task 12750979

Name	hadcm3n_o7d4_1900_40_007204875_0
Workunit	7403155
Created	28 Mar 2011, 14:20:05 UTC
Sent	28 Mar 2011, 16:58:10 UTC
Report deadline	28 Jun 2011, 0:25:21 UTC
Received	31 Mar 2011, 22:05:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1136193
Run time	2 days 4 hours 28 min 43 sec
CPU time	1 days 4 hours 23 min 14 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Tue Mar 29 10:46:39 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x003353a5 _two_coeff_region + 1445 1 hadcm3n_um_6.07_i686-apple-darwin 0x0032cf27 _triple_column + 487 2 hadcm3n_um_6.07_i686-apple-darwin 0x001e7c35 _monochromatic_flux + 741 3 hadcm3n_um_6.07_i686-apple-darwin 0x0029a771 _solve_band_k_eqv + 11425 4 hadcm3n_um_6.07_i686-apple-darwin 0x0010f59c _flux_calc + 8748 5 hadcm3n_um_6.07_i686-apple-darwin 0x002e5762 _r2_swrad + 6066 6 hadcm3n_um_6.07_i686-apple-darwin 0x0024d95d _rad_ctl + 24573 7 hadcm3n_um_6.07_i686-apple-darwin 0x00040abc _atm_phys + 8444 8 hadcm3n_um_6.07_i686-apple-darwin 0x00376f9d _atm_step + 34797 9 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 10 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 11 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 12 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 13 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdc0b988 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57273, iMonCtr=1 Model crash detected, will try to restart... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Tue Mar 29 13:26:34 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x924fd257 _sprintf + 350 1 hadcm3n_um_6.07_i686-apple-darwin 0x0000c18a __ZN10cpdn_shmem12CopyChecksumEPKcS1_ + 130 2 hadcm3n_um_6.07_i686-apple-darwin 0x0002ce1c _fast_tanh. + 255 3 hadcm3n_um_6.07_i686-apple-darwin 0x0019dd0c _isopyc_a + 12700 4 hadcm3n_um_6.07_i686-apple-darwin 0x001a36b3 _isopyc_m + 5843 5 hadcm3n_um_6.07_i686-apple-darwin 0x002fe3b9 _tracer + 14633 6 hadcm3n_um_6.07_i686-apple-darwin 0x0027b525 _rowcalc + 38533 7 hadcm3n_um_6.07_i686-apple-darwin 0x00056cf6 _blokcntl + 12566 8 hadcm3n_um_6.07_i686-apple-darwin 0x0005012b _blokcalc + 16843 9 hadcm3n_um_6.07_i686-apple-darwin 0x00287e62 _row_ctl + 9426 10 hadcm3n_um_6.07_i686-apple-darwin 0x001f9a84 _ocn_ctl + 7252 11 hadcm3n_um_6.07_i686-apple-darwin 0x001f7253 _ocn_for_step + 6755 12 hadcm3n_um_6.07_i686-apple-darwin 0x00380efd _ocn_step + 3693 13 hadcm3n_um_6.07_i686-apple-darwin 0x0039539d _u_model + 45245 14 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 15 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 16 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 17 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbdd91788 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57273, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Tue Mar 29 19:57:17 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x002ee0a6 _th_adv + 8982 1 hadcm3n_um_6.07_i686-apple-darwin 0x0001d234 _adv_ctl + 5780 2 hadcm3n_um_6.07_i686-apple-darwin 0x00038eb0 _atm_dyn + 32144 3 hadcm3n_um_6.07_i686-apple-darwin 0x0037020b _atm_step + 6747 4 hadcm3n_um_6.07_i686-apple-darwin 0x003955ea _u_model + 45834 5 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 6 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 7 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 8 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbe79c848 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=61512, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGBUS: bus error Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.6 build 10J567 Wed Mar 30 00:23:57 2011 Thread 0 Crashed: 0 hadcm3n_um_6.07_i686-apple-darwin 0x0041be30 _expf.J + 192 1 hadcm3n_um_6.07_i686-apple-darwin 0x0002ce1c _fast_tanh. + 255 2 hadcm3n_um_6.07_i686-apple-darwin 0x0002be15 _ai_calc + 32325 3 hadcm3n_um_6.07_i686-apple-darwin 0x001a2fc4 _isopyc_m + 4068 4 hadcm3n_um_6.07_i686-apple-darwin 0x002fe3b9 _tracer + 14633 5 hadcm3n_um_6.07_i686-apple-darwin 0x0027b525 _rowcalc + 38533 6 hadcm3n_um_6.07_i686-apple-darwin 0x00056cf6 _blokcntl + 12566 7 hadcm3n_um_6.07_i686-apple-darwin 0x0005012b _blokcalc + 16843 8 hadcm3n_um_6.07_i686-apple-darwin 0x00287e62 _row_ctl + 9426 9 hadcm3n_um_6.07_i686-apple-darwin 0x001f9a84 _ocn_ctl + 7252 10 hadcm3n_um_6.07_i686-apple-darwin 0x001f7253 _ocn_for_step + 6755 11 hadcm3n_um_6.07_i686-apple-darwin 0x00380efd _ocn_step + 3693 12 hadcm3n_um_6.07_i686-apple-darwin 0x0039539d _u_model + 45245 13 hadcm3n_um_6.07_i686-apple-darwin 0x0038a1ed _um_shell + 5613 14 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 15 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 16 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x924f120e _mach_wait_until + 10 1 libSystem.B.dylib 0x92578819 _nanosleep + 345 2 libSystem.B.dylib 0x925786ba _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9251e85d __pthread_start + 345 6 libSystem.B.dylib 0x9251e6e2 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbddb7358 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x924e1000 - 0x924effff /usr/lib/libz.1.dylib 0x924f0000 - 0x92697fff /usr/lib/libSystem.B.dylib 0x96222000 - 0x96225fff /usr/lib/system/libmathCommon.A.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=63546, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Mar 2011 16:08:11	1136193	12750979	hadcm3n_o7d4_1900_40_007204875_0	77,760	124,369	1.5994
30 Mar 2011 18:36:56	1136193	12750979	hadcm3n_o7d4_1900_40_007204875_0	51,840	78,697	1.5181
29 Mar 2011 15:45:06	1136193	12750979	hadcm3n_o7d4_1900_40_007204875_0	25,920	33,679	1.2993