Task 13604606

Name	hadcm3n_yabg_1980_40_007537672_0
Workunit	7734904
Created	5 Nov 2011, 17:19:09 UTC
Sent	5 Nov 2011, 17:24:28 UTC
Report deadline	5 Feb 2012, 0:51:39 UTC
Received	2 Mar 2012, 18:37:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1121257
Run time	14 days 2 hours 42 min 37 sec
CPU time	13 days 16 hours 48 min 22 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=972, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:00:29 (5400): Can't acquire lockfile (32) - waiting 35s 18:16:34 (2900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yabgko.pji9c10 Error converting file to netcdf: dataout/yabgko.pii9c10 Error converting file to netcdf: dataout/yabgko.pfi9c10 Error converting file to netcdf: dataout/yabgka.phi9c10 Error converting file to netcdf: dataout/yabgka.pgi9c10 Error converting file to netcdf: dataout/yabgka.pei9c10 Error converting file to netcdf: dataout/yabgka.pdi9c10 CPDN Monitor - Quit request from BOINC... 21:11:33 (5636): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4324, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:38:44 (6092): Can't acquire lockfile (32) - waiting 35s Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:01:05 (5640): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... 18:05:30 (5448): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:03:27 (6096): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6096, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:27:24 (4520): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:34:39 (5592): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:24:18 (5584): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:36:07 (5376): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:20 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:21:56 (3752): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2804, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:37:02 (4128): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:01:59 (3308): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:19 (3048): No heartbeat from core client for 30 sec - exiting 18:02:20 (3048): No heartbeat from core client for 30 sec - exiting 18:02:21 (3048): No heartbeat from core client for 30 sec - exiting 18:02:22 (3048): No heartbeat from core client for 30 sec - exiting 18:02:23 (3048): No heartbeat from core client for 30 sec - exiting 18:02:24 (3048): No heartbeat from core client for 30 sec - exiting 18:02:25 (3048): No heartbeat from core client for 30 sec - exiting 18:02:26 (3048): No heartbeat from core client for 30 sec - exiting 18:02:27 (3048): No heartbeat from core client for 30 sec - exiting 18:02:28 (3048): No heartbeat from core client for 30 sec - exiting 18:02:29 (3048): No heartbeat from core client for 30 sec - exiting 18:02:30 (3048): No heartbeat from core client for 30 sec - exiting 18:02:31 (3048): No heartbeat from core client for 30 sec - exiting 18:02:32 (3048): No heartbeat from core client for 30 sec - exiting 18:02:33 (3048): No heartbeat from core client for 30 sec - exiting 18:02:34 (3048): No heartbeat from core client for 30 sec - exiting 18:02:35 (3048): No heartbeat from core client for 30 sec - exiting 18:02:36 (3048): No heartbeat from core client for 30 sec - exiting 18:02:37 (3048): No heartbeat from core client for 30 sec - exiting 18:02:38 (3048): No heartbeat from core client for 30 sec - exiting 18:02:39 (3048): No heartbeat from core client for 30 sec - exiting 18:02:40 (3048): No heartbeat from core client for 30 sec - exiting 18:02:41 (3048): No heartbeat from core client for 30 sec - exiting 18:02:42 (3048): No heartbeat from core client for 30 sec - exiting 18:02:43 (3048): No heartbeat from core client for 30 sec - exiting 18:02:44 (3048): No heartbeat from core client for 30 sec - exiting 18:02:45 (3048): No heartbeat from core client for 30 sec - exiting 18:02:46 (3048): No heartbeat from core client for 30 sec - exiting 18:02:46 (1572): Can't acquire lockfile (32) - waiting 35s 18:02:47 (3048): No heartbeat from core client for 30 sec - exiting 18:02:48 (3048): No heartbeat from core client for 30 sec - exiting 18:02:49 (3048): No heartbeat from core client for 30 sec - exiting 18:02:50 (3048): No heartbeat from core client for 30 sec - exiting 18:02:51 (3048): No heartbeat from core client for 30 sec - exiting 18:02:52 (3048): No heartbeat from core client for 30 sec - exiting 18:02:53 (3048): No heartbeat from core client for 30 sec - exiting 18:02:54 (3048): No heartbeat from core client for 30 sec - exiting 18:02:55 (3048): No heartbeat from core client for 30 sec - exiting 18:02:56 (3048): No heartbeat from core client for 30 sec - exiting 18:02:57 (3048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3948, selfPID=3948, iMonCtr=1 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yabgko.pjl0c10 Error converting file to netcdf: dataout/yabgko.pil0c10 Error converting file to netcdf: dataout/yabgko.pfl0c10 Error converting file to netcdf: dataout/yabgko.pcl0c10 Error converting file to netcdf: dataout/yabgko.pbl0c10 Error converting file to netcdf: dataout/yabgko.pal0c10 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77DE3242 read attempt to address 0x4092AF7E Engaging BOINC Windows Runtime Debugger... Cannot serialize file E:\BOINC/projects/climateprediction.net/hadcm3n_yabg_1980_40_007537672/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Mar 2012 18:08:17	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	777,600	1,183,698	1.5222
20 Feb 2012 19:24:31	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	751,680	1,145,375	1.5238
19 Feb 2012 14:05:07	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	725,760	1,108,102	1.5268
12 Feb 2012 19:23:30	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	699,840	1,069,126	1.5277
10 Feb 2012 18:13:56	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	673,920	1,030,193	1.5287
05 Feb 2012 12:23:20	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	648,000	991,327	1.5298
03 Feb 2012 18:16:28	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	622,080	952,896	1.5318
29 Jan 2012 13:17:36	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	596,160	913,923	1.5330
22 Jan 2012 20:06:56	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	570,240	874,687	1.5339
13 Jan 2012 17:29:40	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	544,320	835,504	1.5350
07 Jan 2012 12:22:58	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	518,400	796,627	1.5367
01 Jan 2012 19:19:47	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	492,480	757,396	1.5379
30 Dec 2011 17:24:43	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	466,560	717,840	1.5386
27 Dec 2011 19:04:46	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	440,640	678,054	1.5388
26 Dec 2011 14:26:12	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	414,720	637,685	1.5376
24 Dec 2011 19:08:45	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	388,800	597,254	1.5361
22 Dec 2011 20:17:14	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	362,880	557,503	1.5363
21 Dec 2011 17:20:16	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	336,960	518,159	1.5377
12 Dec 2011 19:54:29	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	311,040	477,307	1.5346
09 Dec 2011 19:42:22	1121257	13604606	hadcm3n_yabg_1980_40_007537672_0	285,120	437,886	1.5358