Task 11772202

Name	famous_vcwg_1399_200_006702486_3
Workunit	6905739
Created	26 Aug 2010, 16:45:51 UTC
Sent	30 Nov 2010, 12:54:29 UTC
Report deadline	1 Mar 2011, 20:21:40 UTC
Received	10 Dec 2010, 23:02:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	989980
Run time	1 days 1 hours 34 min 54 sec
CPU time	23 hours 44 min 36 sec
Validate state	Invalid
Credit	1,760.34
Device peak FLOPS	3.28 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-apple-darwin
Stderr	<core_client_version>6.10.43</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> (87649): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (88149): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (93436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (94312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (94375): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Thu Dec 9 14:44:08 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x990ae46b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc40c edx: 0x990480fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc448 esp: 0xbfffc40c ss: 0x0000001f efl: 0x00000206 eip: 0x990480fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libirc.dylib 0x90d0c000 - 0x90d0ffff /usr/lib/system/libmathCommon.A.dylib 0x96d8d000 - 0x96d9bfff /usr/lib/libz.1.dylib 0x99047000 - 0x991eefff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=94581, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... (94633): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Thu Dec 9 18:50:43 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x990ae46b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x990480fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c ss: 0x0000001f efl: 0x00000206 eip: 0x990480fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libirc.dylib 0x90d0c000 - 0x90d0ffff /usr/lib/system/libmathCommon.A.dylib 0x96d8d000 - 0x96d9bfff /usr/lib/libz.1.dylib 0x99047000 - 0x991eefff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=97262, iMonCtr=1 Model crash detected, will try to restart... (97262): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Thu Dec 9 19:58:48 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x990ae46b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc40c edx: 0x990480fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc448 esp: 0xbfffc40c ss: 0x0000001f efl: 0x00000206 eip: 0x990480fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libirc.dylib 0x90d0c000 - 0x90d0ffff /usr/lib/system/libmathCommon.A.dylib 0x96d8d000 - 0x96d9bfff /usr/lib/libz.1.dylib 0x99047000 - 0x991eefff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=97965, iMonCtr=1 Model crash detected, will try to restart... (97965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... (2456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (4191): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... (5226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5750): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (11049): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (13030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Fri Dec 10 19:45:23 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x990ae46b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc40c edx: 0x990480fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc448 esp: 0xbfffc40c ss: 0x0000001f efl: 0x00000206 eip: 0x990480fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vcwg_1399_200_006702486/lib/libirc.dylib 0x90d0c000 - 0x90d0ffff /usr/lib/system/libmathCommon.A.dylib 0x96d8d000 - 0x96d9bfff /usr/lib/libz.1.dylib 0x99047000 - 0x991eefff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13615, iMonCtr=1 Model crash detected, will try to restart... (13615): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... (14664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (14859): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (14943): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (15585): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Dec 2010 22:48:30	989980	11772202	famous_vcwg_1399_200_006702486_3	533,546	84,651	0.1587
10 Dec 2010 22:22:38	989980	11772202	famous_vcwg_1399_200_006702486_3	524,186	83,167	0.1587
10 Dec 2010 21:56:38	989980	11772202	famous_vcwg_1399_200_006702486_3	514,826	81,691	0.1587
10 Dec 2010 21:30:34	989980	11772202	famous_vcwg_1399_200_006702486_3	505,466	80,212	0.1587
10 Dec 2010 20:59:08	989980	11772202	famous_vcwg_1399_200_006702486_3	496,106	78,796	0.1588
10 Dec 2010 20:32:40	989980	11772202	famous_vcwg_1399_200_006702486_3	486,746	77,300	0.1588
10 Dec 2010 20:06:32	989980	11772202	famous_vcwg_1399_200_006702486_3	477,386	75,869	0.1589
10 Dec 2010 19:35:17	989980	11772202	famous_vcwg_1399_200_006702486_3	468,026	74,502	0.1592
10 Dec 2010 19:14:18	989980	11772202	famous_vcwg_1399_200_006702486_3	458,666	73,075	0.1593
10 Dec 2010 18:43:09	989980	11772202	famous_vcwg_1399_200_006702486_3	449,306	71,621	0.1594
10 Dec 2010 18:17:11	989980	11772202	famous_vcwg_1399_200_006702486_3	439,946	70,189	0.1595
10 Dec 2010 17:51:07	989980	11772202	famous_vcwg_1399_200_006702486_3	430,586	68,819	0.1598
10 Dec 2010 17:24:50	989980	11772202	famous_vcwg_1399_200_006702486_3	421,226	67,290	0.1597
10 Dec 2010 16:58:42	989980	11772202	famous_vcwg_1399_200_006702486_3	411,866	65,754	0.1596
10 Dec 2010 16:32:30	989980	11772202	famous_vcwg_1399_200_006702486_3	402,506	64,196	0.1595
10 Dec 2010 16:06:13	989980	11772202	famous_vcwg_1399_200_006702486_3	393,146	62,662	0.1594
10 Dec 2010 15:39:45	989980	11772202	famous_vcwg_1399_200_006702486_3	383,786	61,140	0.1593
10 Dec 2010 15:13:35	989980	11772202	famous_vcwg_1399_200_006702486_3	374,426	59,618	0.1592
10 Dec 2010 11:25:17	989980	11772202	famous_vcwg_1399_200_006702486_3	365,066	58,096	0.1591
10 Dec 2010 11:01:27	989980	11772202	famous_vcwg_1399_200_006702486_3	355,706	56,568	0.1590