Task 12896454

Name	hadcm3n_p6er_1900_40_007225891_2
Workunit	7424131
Created	19 May 2011, 1:39:13 UTC
Sent	19 May 2011, 1:39:17 UTC
Report deadline	18 Aug 2011, 9:06:28 UTC
Received	19 Jul 2011, 0:30:10 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1131576
Run time	22 days 3 hours 49 min 54 sec
CPU time	21 days 19 hours 12 min 16 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=820, selfPID=820, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6912, selfPID=6912, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4440, selfPID=4440, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5776, selfPID=5776, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2964, selfPID=2964, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2468, selfPID=2468, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6380, selfPID=6380, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6088, selfPID=6088, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3232, selfPID=3232, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5800, selfPID=5800, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1396, selfPID=1396, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5900, selfPID=5900, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5872, selfPID=5872, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6800, selfPID=6800, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1168, selfPID=1168, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2172, selfPID=2172, iMonCtr=1 CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5804, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5804, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5316, selfPID=5316, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5176, selfPID=5176, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5616, selfPID=5616, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3872, selfPID=3872, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=952, selfPID=952, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4684, selfPID=4684, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4488, selfPID=4488, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7036, selfPID=7036, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5984, selfPID=5984, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3712, selfPID=3712, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5736, selfPID=5736, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4936, selfPID=4936, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6104, selfPID=6104, iMonCtr=1 10:27:02 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:04 (4732): No heartbeat from core client for 30 sec - exiting 10:27:05 (4732): No heartbeat from core client for 30 sec - exiting 10:27:06 (4732): No heartbeat from core client for 30 sec - exiting 10:27:07 (4732): No heartbeat from core client for 30 sec - exiting 10:27:08 (4732): No heartbeat from core client for 30 sec - exiting 10:27:09 (4732): No heartbeat from core client for 30 sec - exiting 10:27:10 (4732): No heartbeat from core client for 30 sec - exiting 10:27:11 (4732): No heartbeat from core client for 30 sec - exiting 10:27:12 (4732): No heartbeat from core client for 30 sec - exiting 10:27:13 (4732): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5104, selfPID=5104, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2760, selfPID=2760, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5892, selfPID=5892, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6156, selfPID=6156, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4892, selfPID=4892, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6056, selfPID=6056, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1028, selfPID=1028, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7084, selfPID=7084, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2816, selfPID=2816, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6976, selfPID=6976, iMonCtr=1 18:10:34 (4876): No heartbeat from core client for 30 sec - exiting 18:10:35 (4876): No heartbeat from core client for 30 sec - exiting 18:10:36 (4876): No heartbeat from core client for 30 sec - exiting 18:10:38 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:39 (4876): No heartbeat from core client for 30 sec - exiting 18:10:40 (4876): No heartbeat from core client for 30 sec - exiting 18:10:41 (4876): No heartbeat from core client for 30 sec - exiting 18:10:42 (4876): No heartbeat from core client for 30 sec - exiting 18:10:43 (4876): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4752, selfPID=4752, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5860, selfPID=5860, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5284, selfPID=5284, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6408, selfPID=6408, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7144, selfPID=7144, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2824, selfPID=2824, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2148, selfPID=2148, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6756, selfPID=6756, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3844, selfPID=3844, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5152, selfPID=5152, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5832, selfPID=5832, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6092, selfPID=6092, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7028, selfPID=7028, iMonCtr=1 CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6516, selfPID=6516, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7128, selfPID=7128, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4920, selfPID=4920, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2740, selfPID=2740, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4460, selfPID=4460, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5612, selfPID=5612, iMonCtr=1 Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 16:43:41	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	1,036,800	1,882,836	1.8160
25 Jul 2011 13:02:51	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	1,010,880	1,834,897	1.8151
25 Jul 2011 13:02:51	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	984,960	1,792,634	1.8200
10 Jul 2011 04:45:24	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	959,040	1,743,556	1.8180
09 Jul 2011 19:31:16	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	933,120	1,692,882	1.8142
08 Jul 2011 14:50:38	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	907,200	1,643,396	1.8115
07 Jul 2011 15:59:22	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	881,280	1,594,618	1.8094
07 Jul 2011 15:42:00	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	855,360	1,546,226	1.8077
07 Jul 2011 15:42:00	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	829,440	1,498,384	1.8065
05 Jul 2011 11:09:07	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	803,520	1,450,625	1.8053
04 Jul 2011 15:38:11	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	777,600	1,402,293	1.8034
04 Jul 2011 01:58:25	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	751,680	1,353,944	1.8012
03 Jul 2011 06:27:18	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	725,760	1,306,274	1.7999
02 Jul 2011 10:52:19	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	699,840	1,257,849	1.7973
01 Jul 2011 01:20:34	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	673,920	1,209,500	1.7947
28 Jun 2011 06:02:23	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	648,000	1,161,592	1.7926
27 Jun 2011 04:42:40	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	622,080	1,114,079	1.7909
25 Jun 2011 05:41:19	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	596,160	1,066,132	1.7883
24 Jun 2011 04:19:13	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	570,240	1,018,406	1.7859
23 Jun 2011 02:44:52	1131576	12896454	hadcm3n_p6er_1900_40_007225891_2	544,320	970,079	1.7822