Task 14323907

Name	hadcm3n_o0hd_1980_40_007618316_3
Workunit	7796446
Created	27 Mar 2012, 7:17:00 UTC
Sent	27 Mar 2012, 7:17:21 UTC
Report deadline	26 Jun 2012, 14:44:32 UTC
Received	16 Apr 2012, 19:43:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1169903
Run time	17 days 10 hours 19 min 33 sec
CPU time	16 days 16 hours 56 min 53 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 03:45:13 (23702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:31 (25862): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:34 (57048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:54 (40594): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:04:48 (63533): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:39:24 (799): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:31:59 (10983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:32 (18023): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:30:28 (88022): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:23 (27564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:51:25 (36670): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:10 (36784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:17 (36883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:53 (37100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:06 (37311): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:28 (37438): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:35 (37557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:27:46 (37672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:58 (37813): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:36:25 (37924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:52 (38052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:44:58 (38187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:48:34 (38279): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:11 (38430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:07 (38521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:18 (38611): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 05:23:26 (81670): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:41 (54194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:54:40 (24874): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 15:59:08 (92140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:53 (32605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:13:50 (32659): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136185) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=34484, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136185) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=34484, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136185) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=34484, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136185) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=810, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136185) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=810, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 136185) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=810, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Apr 2012 15:34:14	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	855,360	1,422,930	1.6635
13 Apr 2012 02:55:49	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	829,440	1,380,324	1.6642
12 Apr 2012 14:25:36	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	803,520	1,337,750	1.6649
12 Apr 2012 01:36:35	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	777,600	1,295,101	1.6655
11 Apr 2012 12:58:30	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	751,680	1,252,765	1.6666
11 Apr 2012 01:03:04	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	725,760	1,209,897	1.6671
10 Apr 2012 12:12:21	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	699,840	1,167,317	1.6680
09 Apr 2012 23:26:39	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	673,920	1,124,938	1.6692
09 Apr 2012 11:37:35	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	648,000	1,082,656	1.6708
08 Apr 2012 22:28:09	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	622,080	1,040,275	1.6723
08 Apr 2012 09:19:22	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	596,160	997,016	1.6724
07 Apr 2012 20:19:42	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	570,240	953,336	1.6718
07 Apr 2012 07:20:24	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	544,320	910,005	1.6718
06 Apr 2012 18:18:57	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	518,400	866,665	1.6718
06 Apr 2012 05:07:07	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	492,480	823,580	1.6723
05 Apr 2012 16:50:36	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	466,560	780,113	1.6721
05 Apr 2012 02:59:42	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	440,640	734,058	1.6659
04 Apr 2012 13:47:35	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	414,720	689,731	1.6631
04 Apr 2012 00:38:32	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	388,800	646,292	1.6623
03 Apr 2012 11:57:47	1169903	14323907	hadcm3n_o0hd_1980_40_007618316_3	362,880	602,829	1.6612