Task 13739671

Name	hadcm3n_yh1c_1940_40_007607859_2
Workunit	7785989
Created	6 Dec 2011, 20:48:30 UTC
Sent	6 Dec 2011, 20:59:27 UTC
Report deadline	7 Mar 2012, 4:26:38 UTC
Received	16 Apr 2012, 17:03:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1121257
Run time	13 days 19 hours 46 min 51 sec
CPU time	13 days 8 hours 12 min 0 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.21 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4444, iMonCtr=1 Model crash detected, will try to restart... 18:38:44 (6124): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yh1cko.pje1c10 Error converting file to netcdf: dataout/yh1cko.pie1c10 Error converting file to netcdf: dataout/yh1cko.pfe1c10 Error converting file to netcdf: dataout/yh1cka.phe1c10 Error converting file to netcdf: dataout/yh1cka.pge1c10 Error converting file to netcdf: dataout/yh1cka.pee1c10 Error converting file to netcdf: dataout/yh1cka.pde1c10 CPDN Monitor - Quit request from BOINC... 03:01:05 (3148): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4908, iMonCtr=1 Model crash detected, will try to restart... 18:05:30 (5480): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:03:27 (6128): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6128, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:34:39 (5628): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:19:33 (4140): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... 19:24:18 (5592): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:36:07 (5384): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:19 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:20 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:21:56 (2648): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:37:02 (5124): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:01:59 (5028): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:19 (2924): No heartbeat from core client for 30 sec - exiting 18:02:20 (2924): No heartbeat from core client for 30 sec - exiting 18:02:21 (2924): No heartbeat from core client for 30 sec - exiting 18:02:22 (2924): No heartbeat from core client for 30 sec - exiting 18:02:23 (2924): No heartbeat from core client for 30 sec - exiting 18:02:24 (2924): No heartbeat from core client for 30 sec - exiting 18:02:25 (2924): No heartbeat from core client for 30 sec - exiting 18:02:26 (2924): No heartbeat from core client for 30 sec - exiting 18:02:27 (2924): No heartbeat from core client for 30 sec - exiting 18:02:28 (2924): No heartbeat from core client for 30 sec - exiting 18:02:29 (2924): No heartbeat from core client for 30 sec - exiting 18:02:30 (2924): No heartbeat from core client for 30 sec - exiting 18:02:31 (2924): No heartbeat from core client for 30 sec - exiting 18:02:32 (2924): No heartbeat from core client for 30 sec - exiting 18:02:33 (2924): No heartbeat from core client for 30 sec - exiting 18:02:34 (2924): No heartbeat from core client for 30 sec - exiting 18:02:35 (2924): No heartbeat from core client for 30 sec - exiting 18:02:36 (2924): No heartbeat from core client for 30 sec - exiting 18:02:37 (2924): No heartbeat from core client for 30 sec - exiting 18:02:38 (2924): No heartbeat from core client for 30 sec - exiting 18:02:39 (2924): No heartbeat from core client for 30 sec - exiting 18:02:40 (2924): No heartbeat from core client for 30 sec - exiting 18:02:41 (2924): No heartbeat from core client for 30 sec - exiting 18:02:42 (2924): No heartbeat from core client for 30 sec - exiting 18:02:43 (2924): No heartbeat from core client for 30 sec - exiting 18:02:44 (2924): No heartbeat from core client for 30 sec - exiting 18:02:45 (2924): No heartbeat from core client for 30 sec - exiting 18:02:46 (2924): No heartbeat from core client for 30 sec - exiting 18:02:46 (4488): Can't acquire lockfile (32) - waiting 35s 18:02:47 (2924): No heartbeat from core client for 30 sec - exiting 18:02:48 (2924): No heartbeat from core client for 30 sec - exiting 18:02:49 (2924): No heartbeat from core client for 30 sec - exiting 18:02:50 (2924): No heartbeat from core client for 30 sec - exiting 18:02:51 (2924): No heartbeat from core client for 30 sec - exiting 18:02:52 (2924): No heartbeat from core client for 30 sec - exiting 18:02:53 (2924): No heartbeat from core client for 30 sec - exiting 18:02:54 (2924): No heartbeat from core client for 30 sec - exiting 18:02:55 (2924): No heartbeat from core client for 30 sec - exiting 18:02:56 (2924): No heartbeat from core client for 30 sec - exiting 18:02:57 (2924): No heartbeat from core client for 30 sec - exiting 18:02:58 (2924): No heartbeat from core client for 30 sec - exiting 18:02:59 (2924): No heartbeat from core client for 30 sec - exiting 18:03:00 (2924): No heartbeat from core client for 30 sec - exiting 18:03:01 (2924): No heartbeat from core client for 30 sec - exiting 18:03:02 (2924): No heartbeat from core client for 30 sec - exiting 18:03:03 (2924): No heartbeat from core client for 30 sec - exiting 18:03:04 (2924): No heartbeat from core client for 30 sec - exiting 18:03:05 (2924): No heartbeat from core client for 30 sec - exiting 18:03:06 (2924): No heartbeat from core client for 30 sec - exiting 18:03:07 (2924): No heartbeat from core client for 30 sec - exiting 18:03:08 (2924): No heartbeat from core client for 30 sec - exiting 18:03:08 (4496): Can't set up shared mem: -1. Will run in standalone mode. 18:03:09 (2924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3644, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:40:05 (4752): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77D43242 read attempt to address 0x40BC7F6F Engaging BOINC Windows Runtime Debugger... Cannot serialize file E:\BOINC/projects/climateprediction.net/hadcm3n_yh1c_1940_40_007607859/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Apr 2012 17:05:53	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	777,600	1,152,715	1.4824
13 Apr 2012 17:55:33	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	751,680	1,118,109	1.4875
09 Apr 2012 13:09:19	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	725,760	1,081,555	1.4902
06 Apr 2012 15:13:17	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	699,840	1,043,717	1.4914
31 Mar 2012 17:02:32	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	673,920	1,005,564	1.4921
26 Mar 2012 17:42:23	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	648,000	967,526	1.4931
19 Mar 2012 19:26:20	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	622,080	929,149	1.4936
18 Mar 2012 10:58:38	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	596,160	890,575	1.4939
11 Mar 2012 13:55:47	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	570,240	852,314	1.4947
08 Mar 2012 18:04:16	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	544,320	813,166	1.4939
26 Feb 2012 11:31:15	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	518,400	774,786	1.4946
19 Feb 2012 21:14:31	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	492,480	738,029	1.4986
16 Feb 2012 16:37:40	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	466,560	700,298	1.5010
12 Feb 2012 15:46:01	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	440,640	661,475	1.5012
08 Feb 2012 17:58:48	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	414,720	623,198	1.5027
04 Feb 2012 17:56:05	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	388,800	584,941	1.5045
01 Feb 2012 18:44:13	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	362,880	546,851	1.5070
28 Jan 2012 22:36:32	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	336,960	508,165	1.5081
18 Jan 2012 18:43:01	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	311,040	469,396	1.5091
11 Jan 2012 17:37:20	1121257	13739671	hadcm3n_yh1c_1940_40_007607859_2	285,120	430,618	1.5103