Task 15290628

Name	hadcm3n_zbwh_1880_40_008199969_2
Workunit	8355093
Created	18 Sep 2012, 3:38:00 UTC
Sent	18 Sep 2012, 3:38:17 UTC
Report deadline	18 Dec 2012, 11:05:28 UTC
Received	14 Dec 2012, 0:10:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1237729
Run time	4 days 5 hours 2 min 46 sec
CPU time	3 days 19 hours 20 min 4 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.20 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zbwhko.pj86c10 Error converting file to netcdf: dataout/zbwhko.pi86c10 Error converting file to netcdf: dataout/zbwhko.pf86c10 Error converting file to netcdf: dataout/zbwhka.ph86c10 Error converting file to netcdf: dataout/zbwhka.pg86c10 Error converting file to netcdf: dataout/zbwhka.pe86c10 Error converting file to netcdf: dataout/zbwhka.pd86c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:29:31 (4356): No heartbeat from core client for 30 sec - exiting 17:29:32 (4356): No heartbeat from core client for 30 sec - exiting 17:29:33 (4356): No heartbeat from core client for 30 sec - exiting 17:29:34 (4356): No heartbeat from core client for 30 sec - exiting 17:29:35 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:29:36 (4356): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 11:19:24 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:25 (4780): No heartbeat from core client for 30 sec - exiting 11:19:26 (4780): No heartbeat from core client for 30 sec - exiting 11:19:27 (4780): No heartbeat from core client for 30 sec - exiting 11:19:28 (4780): No heartbeat from core client for 30 sec - exiting 11:19:29 (4780): No heartbeat from core client for 30 sec - exiting 11:19:30 (4780): No heartbeat from core client for 30 sec - exiting 11:19:31 (4780): No heartbeat from core client for 30 sec - exiting 11:19:32 (4780): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:32:40 (5976): No heartbeat from core client for 30 sec - exiting 21:32:41 (5976): No heartbeat from core client for 30 sec - exiting 21:33:12 (5976): No heartbeat from core client for 30 sec - exiting 21:33:13 (5976): No heartbeat from core client for 30 sec - exiting 21:33:14 (5976): No heartbeat from core client for 30 sec - exiting 21:33:15 (5976): No heartbeat from core client for 30 sec - exiting 21:33:16 (5976): No heartbeat from core client for 30 sec - exiting 21:33:17 (5976): No heartbeat from core client for 30 sec - exiting 21:33:18 (5976): No heartbeat from core client for 30 sec - exiting 21:33:19 (5976): No heartbeat from core client for 30 sec - exiting 21:33:20 (5976): No heartbeat from core client for 30 sec - exiting 21:33:21 (5976): No heartbeat from core client for 30 sec - exiting 21:33:22 (5976): No heartbeat from core client for 30 sec - exiting 21:33:23 (5976): No heartbeat from core client for 30 sec - exiting 21:33:24 (5976): No heartbeat from core client for 30 sec - exiting 21:33:25 (5976): No heartbeat from core client for 30 sec - exiting 21:33:26 (5976): No heartbeat from core client for 30 sec - exiting 21:33:27 (5976): No heartbeat from core client for 30 sec - exiting 21:33:28 (5976): No heartbeat from core client for 30 sec - exiting 21:33:29 (5976): No heartbeat from core client for 30 sec - exiting 21:33:30 (5976): No heartbeat from core client for 30 sec - exiting 21:33:31 (5976): No heartbeat from core client for 30 sec - exiting 21:33:32 (5976): No heartbeat from core client for 30 sec - exiting 21:33:33 (5976): No heartbeat from core client for 30 sec - exiting 21:33:34 (5976): No heartbeat from core client for 30 sec - exiting 21:33:35 (5976): No heartbeat from core client for 30 sec - exiting 21:33:36 (5976): No heartbeat from core client for 30 sec - exiting 21:33:37 (5976): No heartbeat from core client for 30 sec - exiting 21:33:38 (5976): No heartbeat from core client for 30 sec - exiting 21:33:39 (5976): No heartbeat from core client for 30 sec - exiting 21:33:40 (5976): No heartbeat from core client for 30 sec - exiting 21:33:41 (5976): No heartbeat from core client for 30 sec - exiting 21:33:42 (5976): No heartbeat from core client for 30 sec - exiting 21:33:43 (5976): No heartbeat from core client for 30 sec - exiting 21:33:44 (5976): No heartbeat from core client for 30 sec - exiting 21:33:45 (5976): No heartbeat from core client for 30 sec - exiting 21:34:16 (5976): No heartbeat from core client for 30 sec - exiting 21:34:17 (5976): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Dec 2012 00:13:31	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	259,200	328,799	1.2685
14 Dec 2012 00:13:31	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	233,280	309,238	1.3256
14 Dec 2012 00:13:31	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	207,360	285,159	1.3752
15 Nov 2012 13:34:54	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	181,440	249,453	1.3749
12 Oct 2012 16:06:37	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	155,520	213,448	1.3725
26 Sep 2012 12:29:31	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	129,600	177,311	1.3681
22 Sep 2012 21:13:07	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	103,680	141,690	1.3666
21 Sep 2012 13:08:49	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	77,760	106,245	1.3663
21 Sep 2012 01:55:51	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	51,840	70,976	1.3691
19 Sep 2012 13:59:21	1237729	15290628	hadcm3n_zbwh_1880_40_008199969_2	25,920	35,223	1.3589