Task 11433136

Name	famous_u2sn_1599_200_006636466_1
Workunit	6839838
Created	10 Jun 2010, 11:32:06 UTC
Sent	19 Jul 2010, 23:41:58 UTC
Report deadline	19 Oct 2010, 7:09:09 UTC
Received	15 Aug 2010, 0:49:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1062556
Run time	4 days 2 hours 18 min 17 sec
CPU time	3 days 7 hours 36 min 31 sec
Validate state	Invalid
Credit	3,026.49
Device peak FLOPS	3.75 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> (11013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (14835): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (16769): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (26165): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (3501): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (7598): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Thu Jul 22 14:32:02 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc2ec edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc328 esp: 0xbfffc2ec ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9843, iMonCtr=1 Model crash detected, will try to restart... (9843): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (15422): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (16123): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (17156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (17652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (18207): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (20398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23597): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Sat Jul 24 16:14:05 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd3ec edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd428 esp: 0xbfffd3ec ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=243, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (1045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (6253): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (7469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (7823): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (8132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (8692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (11297): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (11532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (13148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (14423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... (763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (3834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (8950): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (9994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (263): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (3243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (3986): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (5474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Wed Jul 28 16:32:16 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd 11 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd37c edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd3b8 esp: 0xbfffd37c ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5834, iMonCtr=1 Model crash detected, will try to restart... (5834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (7382): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (10089): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (234): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Thu Jul 29 17:29:01 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd3fc edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd438 esp: 0xbfffd3fc ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=597, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Thu Jul 29 20:22:47 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc2ec edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc328 esp: 0xbfffc2ec ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=255, iMonCtr=1 Model crash detected, will try to restart... (255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (3482): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (5972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (6251): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Fri Jul 30 18:03:49 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd 11 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd37c edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd3b8 esp: 0xbfffd37c ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7405, iMonCtr=1 Model crash detected, will try to restart... (7405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (8137): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (10972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Mon Aug 9 13:46:13 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd3fc edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd438 esp: 0xbfffd3fc ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=746, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Mon Aug 9 16:00:55 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd 11 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd38c edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd3c8 esp: 0xbfffd38c ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1028, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Tue Aug 10 16:34:09 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd3ec edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd428 esp: 0xbfffd3ec ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5202, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (1288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Wed Aug 11 17:23:48 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9442d1fb 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc36c edx: 0x943c70fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc3a8 esp: 0xbfffc36c ss: 0x0000001f efl: 0x00000206 eip: 0x943c70fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_u2sn_1599_200_006636466/lib/libirc.dylib 0x916b4000 - 0x916b7fff /usr/lib/system/libmathCommon.A.dylib 0x943c6000 - 0x9456cfff /usr/lib/libSystem.B.dylib 0x9456d000 - 0x9457bfff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3478, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... (839): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... (2437): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (2288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Sorry, too many model crashes! :-( (244): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Aug 2010 01:28:27	1062556	11433136	famous_u2sn_1599_200_006636466_1	917,306	286,422	0.3122
15 Aug 2010 01:28:27	1062556	11433136	famous_u2sn_1599_200_006636466_1	907,946	285,060	0.3140
14 Aug 2010 19:56:31	1062556	11433136	famous_u2sn_1599_200_006636466_1	898,586	283,711	0.3157
14 Aug 2010 15:29:19	1062556	11433136	famous_u2sn_1599_200_006636466_1	889,226	282,363	0.3175
14 Aug 2010 15:01:58	1062556	11433136	famous_u2sn_1599_200_006636466_1	879,866	281,013	0.3194
14 Aug 2010 13:21:37	1062556	11433136	famous_u2sn_1599_200_006636466_1	870,506	279,645	0.3212
14 Aug 2010 01:37:52	1062556	11433136	famous_u2sn_1599_200_006636466_1	861,146	278,329	0.3232
13 Aug 2010 20:32:53	1062556	11433136	famous_u2sn_1599_200_006636466_1	851,786	276,961	0.3252
13 Aug 2010 19:59:33	1062556	11433136	famous_u2sn_1599_200_006636466_1	842,426	275,595	0.3271
13 Aug 2010 19:32:41	1062556	11433136	famous_u2sn_1599_200_006636466_1	833,066	274,241	0.3292
13 Aug 2010 18:53:12	1062556	11433136	famous_u2sn_1599_200_006636466_1	823,706	272,873	0.3313
13 Aug 2010 18:17:01	1062556	11433136	famous_u2sn_1599_200_006636466_1	814,346	271,494	0.3334
13 Aug 2010 17:45:09	1062556	11433136	famous_u2sn_1599_200_006636466_1	804,986	270,131	0.3356
26 Jul 2010 23:53:08	1062556	11433136	famous_u2sn_1599_200_006636466_1	795,626	154,197	0.1938
26 Jul 2010 23:53:08	1062556	11433136	famous_u2sn_1599_200_006636466_1	786,266	152,894	0.1945
26 Jul 2010 23:53:08	1062556	11433136	famous_u2sn_1599_200_006636466_1	776,906	151,580	0.1951
26 Jul 2010 21:34:24	1062556	11433136	famous_u2sn_1599_200_006636466_1	767,546	150,283	0.1958
26 Jul 2010 21:08:07	1062556	11433136	famous_u2sn_1599_200_006636466_1	758,186	148,976	0.1965
26 Jul 2010 20:47:32	1062556	11433136	famous_u2sn_1599_200_006636466_1	748,826	147,674	0.1972
26 Jul 2010 20:26:37	1062556	11433136	famous_u2sn_1599_200_006636466_1	739,466	146,374	0.1979