Task 12735557

Name	hadcm3n_o1fl_1900_40_007197188_0
Workunit	7395468
Created	28 Mar 2011, 14:00:13 UTC
Sent	1 Apr 2011, 20:23:30 UTC
Report deadline	2 Jul 2011, 3:50:41 UTC
Received	2 Jun 2011, 12:25:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1044015
Run time	13 days 1 hours 26 min 56 sec
CPU time	11 days 18 hours 54 min 55 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sat May 21 06:19:35 2011 Thread 0 Crashed: CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Tue May 24 19:50:08 2011 Thread 0 Crashed: CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sun May 29 17:30:25 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x977e4156 ___kill + 10 1 libSystem.B.dylib 0x97876899 _raise + 26 2 libSystem.B.dylib 0x9788c9b8 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x977831ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9780a809 _nanosleep + 345 2 libSystem.B.dylib 0x9780a6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x977b07fd __pthread_start + 345 6 libSystem.B.dylib 0x977b0682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9266b000 - 0x9266efff /usr/lib/system/libmathCommon.A.dylib 0x96585000 - 0x96593fff /usr/lib/libz.1.dylib 0x97782000 - 0x97929fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38952, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sun May 29 17:30:27 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x977e4156 ___kill + 10 1 libSystem.B.dylib 0x97876899 _raise + 26 2 libSystem.B.dylib 0x9788c9b8 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x977831ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9780a809 _nanosleep + 345 2 libSystem.B.dylib 0x9780a6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x977b07fd __pthread_start + 345 6 libSystem.B.dylib 0x977b0682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9266b000 - 0x9266efff /usr/lib/system/libmathCommon.A.dylib 0x96585000 - 0x96593fff /usr/lib/libz.1.dylib 0x97782000 - 0x97929fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38952, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Sun May 29 17:30:30 2011 Thread 0 Crashed: CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Thu Jun 2 21:48:19 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x977e4156 ___kill + 10 1 libSystem.B.dylib 0x97876899 _raise + 26 2 libSystem.B.dylib 0x9788c9b8 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x977831ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9780a809 _nanosleep + 345 2 libSystem.B.dylib 0x9780a6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x977b07fd __pthread_start + 345 6 libSystem.B.dylib 0x977b0682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9266b000 - 0x9266efff /usr/lib/system/libmathCommon.A.dylib 0x96585000 - 0x96593fff /usr/lib/libz.1.dylib 0x97782000 - 0x97929fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=242, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Thu Jun 2 21:48:24 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x977e4156 ___kill + 10 1 libSystem.B.dylib 0x97876899 _raise + 26 2 libSystem.B.dylib 0x9788c9b8 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x977831ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9780a809 _nanosleep + 345 2 libSystem.B.dylib 0x9780a6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x977b07fd __pthread_start + 345 6 libSystem.B.dylib 0x977b0682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9266b000 - 0x9266efff /usr/lib/system/libmathCommon.A.dylib 0x96585000 - 0x96593fff /usr/lib/libz.1.dylib 0x97782000 - 0x97929fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=242, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Thu Jun 2 21:48:28 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x977e4156 ___kill + 10 1 libSystem.B.dylib 0x97876899 _raise + 26 2 libSystem.B.dylib 0x9788c9b8 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x977831ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9780a809 _nanosleep + 345 2 libSystem.B.dylib 0x9780a6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x977b07fd __pthread_start + 345 6 libSystem.B.dylib 0x977b0682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9266b000 - 0x9266efff /usr/lib/system/libmathCommon.A.dylib 0x96585000 - 0x96593fff /usr/lib/libz.1.dylib 0x97782000 - 0x97929fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=242, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Thu Jun 2 21:48:31 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x977e4156 ___kill + 10 1 libSystem.B.dylib 0x97876899 _raise + 26 2 libSystem.B.dylib 0x9788c9b8 _abort + 93 3 hadcm3n_um_6.07_i686-apple-darwin 0x00336cff _um_setup + 127 4 hadcm3n_um_6.07_i686-apple-darwin 0x00389b71 _um_shell + 3953 5 hadcm3n_um_6.07_i686-apple-darwin 0x000048e1 _main + 1153 6 hadcm3n_um_6.07_i686-apple-darwin 0x000027fb __start + 209 7 hadcm3n_um_6.07_i686-apple-darwin 0x00002729 start + 41 Thread 1: 0 libSystem.B.dylib 0x977831ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9780a809 _nanosleep + 345 2 libSystem.B.dylib 0x9780a6aa _usleep + 61 3 hadcm3n_um_6.07_i686-apple-darwin 0x00442ec0 __Z11boinc_sleepd + 188 4 hadcm3n_um_6.07_i686-apple-darwin 0x0043865e __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x977b07fd __pthread_start + 345 6 libSystem.B.dylib 0x977b0682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca18 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9266b000 - 0x9266efff /usr/lib/system/libmathCommon.A.dylib 0x96585000 - 0x96593fff /usr/lib/libz.1.dylib 0x97782000 - 0x97929fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=242, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 May 2011 23:15:25	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	596,160	995,252	1.6694
18 May 2011 07:28:45	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	570,240	952,257	1.6699
17 May 2011 07:18:50	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	544,320	907,845	1.6679
10 May 2011 00:15:23	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	518,400	865,146	1.6689
07 May 2011 23:47:49	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	492,480	822,166	1.6694
06 May 2011 06:25:16	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	466,560	779,653	1.6711
05 May 2011 12:32:48	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	440,640	736,981	1.6725
04 May 2011 10:59:04	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	414,720	694,282	1.6741
01 May 2011 22:34:02	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	388,800	650,162	1.6722
26 Apr 2011 13:40:52	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	362,880	607,280	1.6735
25 Apr 2011 03:37:12	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	336,960	565,366	1.6778
21 Apr 2011 18:42:27	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	311,040	522,006	1.6783
20 Apr 2011 20:50:59	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	285,120	477,543	1.6749
20 Apr 2011 20:50:59	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	259,200	433,107	1.6709
20 Apr 2011 20:50:59	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	233,280	389,773	1.6708
09 Apr 2011 05:08:33	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	207,360	347,439	1.6755
08 Apr 2011 04:20:36	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	181,440	305,765	1.6852
06 Apr 2011 15:36:49	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	155,520	263,487	1.6942
05 Apr 2011 23:29:14	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	129,600	220,101	1.6983
05 Apr 2011 10:00:10	1044015	12735557	hadcm3n_o1fl_1900_40_007197188_0	103,680	176,076	1.6983