Task 16273782

Name	hadcm3n_80t5_1980_40_008458988_2
Workunit	8609844
Created	18 Jan 2014, 23:22:28 UTC
Sent	18 Jan 2014, 23:22:38 UTC
Report deadline	20 Apr 2014, 6:49:49 UTC
Received	31 Mar 2014, 21:16:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	959076
Run time	56 days 1 hours 9 min 30 sec
CPU time	51 days 1 hours 19 min 15 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	1.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:16:20 (176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:30 (198): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:36 (198): No heartbeat from core client for 30 sec - exiting 17:10:37 (198): No heartbeat from core client for 30 sec - exiting 17:10:38 (198): No heartbeat from core client for 30 sec - exiting 17:10:39 (198): No heartbeat from core client for 30 sec - exiting 17:10:40 (198): No heartbeat from core client for 30 sec - exiting 17:10:41 (198): No heartbeat from core client for 30 sec - exiting 17:10:42 (198): No heartbeat from core client for 30 sec - exiting 17:10:43 (198): No heartbeat from core client for 30 sec - exiting 17:10:44 (198): No heartbeat from core client for 30 sec - exiting 17:10:45 (198): No heartbeat from core client for 30 sec - exiting 17:10:46 (198): No heartbeat from core client for 30 sec - exiting 17:10:47 (198): No heartbeat from core client for 30 sec - exiting 17:10:48 (198): No heartbeat from core client for 30 sec - exiting 17:10:49 (198): No heartbeat from core client for 30 sec - exiting 17:10:50 (198): No heartbeat from core client for 30 sec - exiting 17:10:51 (198): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:29:49 (82160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:58:51 (87808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:53 (87808): No heartbeat from core client for 30 sec - exiting 17:58:54 (87808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:06:16 (89674): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:17 (89674): No heartbeat from core client for 30 sec - exiting 22:06:18 (89674): No heartbeat from core client for 30 sec - exiting 22:06:19 (89674): No heartbeat from core client for 30 sec - exiting 22:06:20 (89674): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:11:59 (90587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:01 (90587): No heartbeat from core client for 30 sec - exiting 00:12:02 (90587): No heartbeat from core client for 30 sec - exiting 00:12:03 (90587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:33:00 (97332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:00:00 (17884): No heartbeat from core client for 30 sec - exiting 21:00:02 (17884): No heartbeat from core client for 30 sec - exiting 21:00:03 (17884): No heartbeat from core client for 30 sec - exiting 21:00:04 (17884): No heartbeat from core client for 30 sec - exiting 21:00:05 (17884): No heartbeat from core client for 30 sec - exiting 21:00:06 (17884): No heartbeat from core client for 30 sec - exiting 21:00:07 (17884): No heartbeat from core client for 30 sec - exiting 21:00:08 (17884): No heartbeat from core client for 30 sec - exiting 21:00:09 (17884): No heartbeat from core client for 30 sec - exiting 21:00:10 (17884): No heartbeat from core client for 30 sec - exiting 21:00:11 (17884): No heartbeat from core client for 30 sec - exiting 21:00:12 (17884): No heartbeat from core client for 30 sec - exiting 21:00:13 (17884): No heartbeat from core client for 30 sec - exiting 21:00:14 (17884): No heartbeat from core client for 30 sec - exiting 21:00:15 (17884): No heartbeat from core client for 30 sec - exiting 21:00:16 (17884): No heartbeat from core client for 30 sec - exiting 21:00:17 (17884): No heartbeat from core client for 30 sec - exiting 21:00:18 (17884): No heartbeat from core client for 30 sec - exiting 21:00:19 (17884): No heartbeat from core client for 30 sec - exiting 21:00:20 (17884): No heartbeat from core client for 30 sec - exiting 21:00:21 (17884): No heartbeat from core client for 30 sec - exiting 21:00:22 (17884): No heartbeat from core client for 30 sec - exiting 21:00:23 (17884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:55:57 (12692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:36 (12709): No heartbeat from core client for 30 sec - exiting 21:56:37 (12709): No heartbeat from core client for 30 sec - exiting 21:56:38 (12709): No heartbeat from core client for 30 sec - exiting 21:56:39 (12709): No heartbeat from core client for 30 sec - exiting 21:56:40 (12709): No heartbeat from core client for 30 sec - exiting 21:56:41 (12709): No heartbeat from core client for 30 sec - exiting 21:56:42 (12709): No heartbeat from core client for 30 sec - exiting 21:56:43 (12709): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:53:50 (13264): No heartbeat from core client for 30 sec - exiting 08:53:51 (13264): No heartbeat from core client for 30 sec - exiting 08:53:52 (13264): No heartbeat from core client for 30 sec - exiting 08:53:53 (13264): No heartbeat from core client for 30 sec - exiting 08:53:54 (13264): No heartbeat from core client for 30 sec - exiting 08:53:55 (13264): No heartbeat from core client for 30 sec - exiting 08:53:56 (13264): No heartbeat from core client for 30 sec - exiting 08:53:57 (13264): No heartbeat from core client for 30 sec - exiting 10:08:32 (18047): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:08:34 (18047): No heartbeat from core client for 30 sec - exiting 10:08:35 (18047): No heartbeat from core client for 30 sec - exiting 10:08:36 (18047): No heartbeat from core client for 30 sec - exiting 10:08:37 (18047): No heartbeat from core client for 30 sec - exiting 10:08:38 (18047): No heartbeat from core client for 30 sec - exiting 10:08:39 (18047): No heartbeat from core client for 30 sec - exiting 10:08:40 (18047): No heartbeat from core client for 30 sec - exiting 10:08:41 (18047): No heartbeat from core client for 30 sec - exiting 10:08:42 (18047): No heartbeat from core client for 30 sec - exiting 10:08:43 (18047): No heartbeat from core client for 30 sec - exiting 10:08:44 (18047): No heartbeat from core client for 30 sec - exiting 10:08:45 (18047): No heartbeat from core client for 30 sec - exiting 10:08:46 (18047): No heartbeat from core client for 30 sec - exiting 10:08:47 (18047): No heartbeat from core client for 30 sec - exiting 10:08:48 (18047): No heartbeat from core client for 30 sec - exiting 10:08:49 (18047): No heartbeat from core client for 30 sec - exiting 10:08:50 (18047): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:10:12 (32924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:14 (32924): No heartbeat from core client for 30 sec - exiting 20:10:15 (32924): No heartbeat from core client for 30 sec - exiting 20:10:16 (32924): No heartbeat from core client for 30 sec - exiting 20:10:17 (32924): No heartbeat from core client for 30 sec - exiting 20:10:18 (32924): No heartbeat from core client for 30 sec - exiting 20:10:19 (32924): No heartbeat from core client for 30 sec - exiting 20:10:20 (32924): No heartbeat from core client for 30 sec - exiting 20:10:21 (32924): No heartbeat from core client for 30 sec - exiting 20:10:22 (32924): No heartbeat from core client for 30 sec - exiting 20:10:23 (32924): No heartbeat from core client for 30 sec - exiting 20:10:24 (32924): No heartbeat from core client for 30 sec - exiting 20:10:25 (32924): No heartbeat from core client for 30 sec - exiting 20:10:26 (32924): No heartbeat from core client for 30 sec - exiting 20:10:27 (32924): No heartbeat from core client for 30 sec - exiting 20:10:28 (32924): No heartbeat from core client for 30 sec - exiting 20:10:29 (32924): No heartbeat from core client for 30 sec - exiting 20:10:30 (32924): No heartbeat from core client for 30 sec - exiting 20:10:31 (32924): No heartbeat from core client for 30 sec - exiting 20:10:32 (32924): No heartbeat from core client for 30 sec - exiting 20:10:33 (32924): No heartbeat from core client for 30 sec - exiting 20:10:34 (32924): No heartbeat from core client for 30 sec - exiting 20:10:35 (32924): No heartbeat from core client for 30 sec - exiting 20:10:36 (32924): No heartbeat from core client for 30 sec - exiting 20:10:37 (32924): No heartbeat from core client for 30 sec - exiting 20:10:38 (32924): No heartbeat from core client for 30 sec - exiting 20:10:39 (32924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Mar 30 17:13:35 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x97b83c5a ___kill + 10 1 libSystem.B.dylib 0x97c165a5 _raise + 26 2 libSystem.B.dylib 0x97c2c6e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x97b22c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x97baa429 _nanosleep + 345 2 libSystem.B.dylib 0x97baa2ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x97b50259 __pthread_start + 345 6 libSystem.B.dylib 0x97b500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc968 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92a7a000 - 0x92a88fff /usr/lib/libz.1.dylib 0x94efb000 - 0x94efefff /usr/lib/system/libmathCommon.A.dylib 0x97b22000 - 0x97cc9fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Mar 30 17:15:05 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x97b83c5a ___kill + 10 1 libSystem.B.dylib 0x97c165a5 _raise + 26 2 libSystem.B.dylib 0x97c2c6e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x97b22c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x97baa429 _nanosleep + 345 2 libSystem.B.dylib 0x97baa2ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x97b50259 __pthread_start + 345 6 libSystem.B.dylib 0x97b500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc968 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92a7a000 - 0x92a88fff /usr/lib/libz.1.dylib 0x94efb000 - 0x94efefff /usr/lib/system/libmathCommon.A.dylib 0x97b22000 - 0x97cc9fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Mar 30 17:15:36 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x97b83c5a ___kill + 10 1 libSystem.B.dylib 0x97c165a5 _raise + 26 2 libSystem.B.dylib 0x97c2c6e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x97b22c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x97baa429 _nanosleep + 345 2 libSystem.B.dylib 0x97baa2ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x97b50259 __pthread_start + 345 6 libSystem.B.dylib 0x97b500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc968 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92a7a000 - 0x92a88fff /usr/lib/libz.1.dylib 0x94efb000 - 0x94efefff /usr/lib/system/libmathCommon.A.dylib 0x97b22000 - 0x97cc9fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Mar 30 17:15:43 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x97b83c5a ___kill + 10 1 libSystem.B.dylib 0x97c165a5 _raise + 26 2 libSystem.B.dylib 0x97c2c6e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x97b22c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x97baa429 _nanosleep + 345 2 libSystem.B.dylib 0x97baa2ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x97b50259 __pthread_start + 345 6 libSystem.B.dylib 0x97b500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc968 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92a7a000 - 0x92a88fff /usr/lib/libz.1.dylib 0x94efb000 - 0x94efefff /usr/lib/system/libmathCommon.A.dylib 0x97b22000 - 0x97cc9fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Mar 30 17:15:50 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x97b83c5a ___kill + 10 1 libSystem.B.dylib 0x97c165a5 _raise + 26 2 libSystem.B.dylib 0x97c2c6e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x97b22c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x97baa429 _nanosleep + 345 2 libSystem.B.dylib 0x97baa2ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x97b50259 __pthread_start + 345 6 libSystem.B.dylib 0x97b500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc968 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92a7a000 - 0x92a88fff /usr/lib/libz.1.dylib 0x94efb000 - 0x94efefff /usr/lib/system/libmathCommon.A.dylib 0x97b22000 - 0x97cc9fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Mar 30 17:15:55 2014 Thread 0 Crashed: 0 libSystem.B.dylib 0x97b83c5a ___kill + 10 1 libSystem.B.dylib 0x97c165a5 _raise + 26 2 libSystem.B.dylib 0x97c2c6e4 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x97b22c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x97baa429 _nanosleep + 345 2 libSystem.B.dylib 0x97baa2ca _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x97b50259 __pthread_start + 345 6 libSystem.B.dylib 0x97b500de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc968 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x92a7a000 - 0x92a88fff /usr/lib/libz.1.dylib 0x94efb000 - 0x94efefff /usr/lib/system/libmathCommon.A.dylib 0x97b22000 - 0x97cc9fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Mar 2014 16:23:42	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	1,010,880	4,318,853	4.2724
27 Mar 2014 00:33:00	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	984,960	4,204,557	4.2688
24 Mar 2014 06:02:24	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	959,040	4,092,626	4.2674
22 Mar 2014 08:34:59	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	933,120	3,979,659	4.2649
20 Mar 2014 19:27:56	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	907,200	3,866,706	4.2622
18 Mar 2014 12:53:52	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	881,280	3,755,370	4.2613
16 Mar 2014 16:34:49	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	855,360	3,644,923	4.2613
13 Mar 2014 16:23:04	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	829,440	3,534,217	4.2610
12 Mar 2014 05:32:04	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	803,520	3,421,922	4.2587
09 Mar 2014 20:10:50	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	777,600	3,308,107	4.2543
07 Mar 2014 15:15:14	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	751,680	3,199,575	4.2566
06 Mar 2014 04:56:20	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	725,760	3,086,901	4.2533
04 Mar 2014 06:33:56	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	699,840	2,979,231	4.2570
02 Mar 2014 00:31:57	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	673,920	2,869,490	4.2579
28 Feb 2014 11:51:42	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	648,000	2,757,329	4.2551
26 Feb 2014 14:04:53	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	622,080	2,647,694	4.2562
24 Feb 2014 17:41:46	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	596,160	2,539,051	4.2590
23 Feb 2014 06:50:37	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	570,240	2,423,113	4.2493
21 Feb 2014 16:49:18	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	544,320	2,310,073	4.2440
20 Feb 2014 00:45:32	959076	16273782	hadcm3n_80t5_1980_40_008458988_2	518,400	2,201,225	4.2462