Task 14105150

Name	hadcm3n_o41k_1940_40_007753721_0
Workunit	7908830
Created	17 Feb 2012, 11:55:23 UTC
Sent	17 Feb 2012, 11:55:38 UTC
Report deadline	18 May 2012, 19:22:49 UTC
Received	10 Apr 2012, 2:07:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-177 (0xFFFFFF4F) ERR_RSC_LIMIT_EXCEEDED
Computer ID	1187145
Run time	16 days 11 hours 3 min 48 sec
CPU time	14 days 12 hours 8 min 30 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	3.20 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.36</core_client_version> <![CDATA[ <message> Maximum memory exceeded </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4132, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4748, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4276, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3824, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7932, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5960, selfPID=5960, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5080, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4564, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5752, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o41kko.pjf9c10 Error converting file to netcdf: dataout/o41kko.pif9c10 Error converting file to netcdf: dataout/o41kko.pff9c10 Error converting file to netcdf: dataout/o41kka.phf9c10 Error converting file to netcdf: dataout/o41kka.pgf9c10 Error converting file to netcdf: dataout/o41kka.pef9c10 Error converting file to netcdf: dataout/o41kka.pdf9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CWorker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4688, selfPID=4688, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:08:12 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5956, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Abort request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Apr 2012 01:05:34	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	829,440	1,251,093	1.5084
05 Apr 2012 05:22:14	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	803,520	1,212,577	1.5091
03 Apr 2012 06:42:04	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	777,600	1,173,936	1.5097
02 Apr 2012 06:46:21	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	751,680	1,135,589	1.5107
01 Apr 2012 08:14:03	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	725,760	1,097,727	1.5125
29 Mar 2012 12:30:46	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	699,840	1,058,175	1.5120
28 Mar 2012 12:52:10	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	673,920	1,018,519	1.5113
27 Mar 2012 10:52:05	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	648,000	978,171	1.5095
24 Mar 2012 12:14:58	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	622,080	939,820	1.5108
22 Mar 2012 23:45:44	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	596,160	901,843	1.5128
21 Mar 2012 11:33:07	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	570,240	861,937	1.5115
21 Mar 2012 00:05:09	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	544,320	822,996	1.5120
20 Mar 2012 00:24:29	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	518,400	785,355	1.5150
19 Mar 2012 03:04:05	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	492,480	747,030	1.5169
16 Mar 2012 04:55:44	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	466,560	709,132	1.5199
14 Mar 2012 10:46:46	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	440,640	670,090	1.5207
12 Mar 2012 00:42:08	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	414,720	630,658	1.5207
11 Mar 2012 10:36:55	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	388,800	589,340	1.5158
10 Mar 2012 14:54:40	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	362,880	548,323	1.5110
09 Mar 2012 19:32:33	1187145	14105150	hadcm3n_o41k_1940_40_007753721_0	336,960	508,895	1.5103