Task 13602546

Name	hadcm3n_y9ue_1980_40_007536848_1
Workunit	7734080
Created	5 Nov 2011, 14:02:38 UTC
Sent	5 Nov 2011, 14:03:36 UTC
Report deadline	4 Feb 2012, 21:30:47 UTC
Received	25 Feb 2012, 18:47:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1121257
Run time	13 days 23 hours 33 min 25 sec
CPU time	13 days 12 hours 4 min 8 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:24:06 (4108): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:57:10 (5544): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:00:29 (5388): Can't acquire lockfile (32) - waiting 35s 18:16:34 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3408, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2580, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:11:33 (5628): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5628, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:23:38 (5952): Can't set up shared mem: -1. Will run in standalone mode. No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4952, iMonCtr=1 Model crash detected, will try to restart... 18:38:44 (6084): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2576, iMonCtr=1 Model crash detected, will try to restart... 18:05:30 (5440): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:03:27 (6084): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4504, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:27:24 (6140): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:34:39 (5584): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/y9ueko.pjk2c10 Error converting file to netcdf: dataout/y9ueko.pik2c10 Error converting file to netcdf: dataout/y9ueko.pfk2c10 Error converting file to netcdf: dataout/y9ueka.phk2c10 Error converting file to netcdf: dataout/y9ueka.pgk2c10 Error converting file to netcdf: dataout/y9ueka.pek2c10 Error converting file to netcdf: dataout/y9ueka.pdk2c10 CPDN Monitor - Quit request from BOINC... 19:19:33 (4972): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... 19:24:18 (5576): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:36:07 (5368): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:19 (4432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:21:56 (4712): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:37:02 (968): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:01:59 (3788): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:19 (4940): No heartbeat from core client for 30 sec - exiting 18:02:20 (4940): No heartbeat from core client for 30 sec - exiting 18:02:21 (4940): No heartbeat from core client for 30 sec - exiting 18:02:22 (4940): No heartbeat from core client for 30 sec - exiting 18:02:23 (4940): No heartbeat from core client for 30 sec - exiting 18:02:24 (4940): No heartbeat from core client for 30 sec - exiting 18:02:25 (4940): No heartbeat from core client for 30 sec - exiting 18:02:26 (4940): No heartbeat from core client for 30 sec - exiting 18:02:27 (4940): No heartbeat from core client for 30 sec - exiting 18:02:28 (4940): No heartbeat from core client for 30 sec - exiting 18:02:29 (4940): No heartbeat from core client for 30 sec - exiting 18:02:30 (4940): No heartbeat from core client for 30 sec - exiting 18:02:31 (4940): No heartbeat from core client for 30 sec - exiting 18:02:32 (4940): No heartbeat from core client for 30 sec - exiting 18:02:33 (4940): No heartbeat from core client for 30 sec - exiting 18:02:34 (4940): No heartbeat from core client for 30 sec - exiting 18:02:35 (4940): No heartbeat from core client for 30 sec - exiting 18:02:36 (4940): No heartbeat from core client for 30 sec - exiting 18:02:37 (4940): No heartbeat from core client for 30 sec - exiting 18:02:38 (4940): No heartbeat from core client for 30 sec - exiting 18:02:39 (4940): No heartbeat from core client for 30 sec - exiting 18:02:40 (4940): No heartbeat from core client for 30 sec - exiting 18:02:41 (4940): No heartbeat from core client for 30 sec - exiting 18:02:42 (4940): No heartbeat from core client for 30 sec - exiting 18:02:43 (4940): No heartbeat from core client for 30 sec - exiting 18:02:44 (4940): No heartbeat from core client for 30 sec - exiting 18:02:45 (4940): No heartbeat from core client for 30 sec - exiting 18:02:46 (4940): No heartbeat from core client for 30 sec - exiting 18:02:46 (2572): Can't acquire lockfile (32) - waiting 35s 18:02:47 (4940): No heartbeat from core client for 30 sec - exiting 18:02:48 (4940): No heartbeat from core client for 30 sec - exiting 18:02:49 (4940): No heartbeat from core client for 30 sec - exiting 18:02:50 (4940): No heartbeat from core client for 30 sec - exiting 18:02:51 (4940): No heartbeat from core client for 30 sec - exiting 18:02:52 (4940): No heartbeat from core client for 30 sec - exiting 18:02:53 (4940): No heartbeat from core client for 30 sec - exiting 18:02:54 (4940): No heartbeat from core client for 30 sec - exiting 18:02:55 (4940): No heartbeat from core client for 30 sec - exiting 18:02:56 (4940): No heartbeat from core client for 30 sec - exiting 18:02:57 (4940): No heartbeat from core client for 30 sec - exiting 18:02:58 (4940): No heartbeat from core client for 30 sec - exiting 18:02:59 (4940): No heartbeat from core client for 30 sec - exiting 18:03:00 (4940): No heartbeat from core client for 30 sec - exiting 18:03:01 (4940): No heartbeat from core client for 30 sec - exiting 18:03:02 (4940): No heartbeat from core client for 30 sec - exiting 18:03:03 (4940): No heartbeat from core client for 30 sec - exiting 18:03:04 (4940): No heartbeat from core client for 30 sec - exiting 18:03:05 (4940): No heartbeat from core client for 30 sec - exiting 18:03:06 (4940): No heartbeat from core client for 30 sec - exiting 18:03:07 (4940): No heartbeat from core client for 30 sec - exiting 18:03:08 (4940): No heartbeat from core client for 30 sec - exiting 18:03:09 (4940): No heartbeat from core client for 30 sec - exiting 18:03:10 (4940): No heartbeat from core client for 30 sec - exiting 18:03:11 (4940): No heartbeat from core client for 30 sec - exiting 18:03:12 (4940): No heartbeat from core client for 30 sec - exiting 18:03:13 (4940): No heartbeat from core client for 30 sec - exiting 18:03:14 (4940): No heartbeat from core client for 30 sec - exiting 18:03:15 (1336): Can't set up shared mem: -1. Will run in standalone mode. 18:03:15 (4940): No heartbeat from core client for 30 sec - exiting 18:03:16 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x773D3709 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file E:\BOINC/projects/climateprediction.net/hadcm3n_y9ue_1980_40_007536848/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Feb 2012 19:16:23	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	777,600	1,166,640	1.5003
19 Feb 2012 21:14:31	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	751,680	1,130,337	1.5037
16 Feb 2012 16:37:40	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	725,760	1,092,549	1.5054
12 Feb 2012 15:46:01	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	699,840	1,053,870	1.5059
08 Feb 2012 17:58:48	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	673,920	1,015,799	1.5073
04 Feb 2012 17:56:04	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	648,000	977,601	1.5086
01 Feb 2012 18:44:12	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	622,080	939,690	1.5106
28 Jan 2012 17:43:44	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	596,160	900,897	1.5112
18 Jan 2012 18:43:01	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	570,240	862,438	1.5124
11 Jan 2012 17:37:20	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	544,320	823,718	1.5133
04 Jan 2012 17:50:14	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	518,400	785,478	1.5152
01 Jan 2012 16:39:06	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	492,480	746,569	1.5159
30 Dec 2011 14:12:38	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	466,560	707,576	1.5166
27 Dec 2011 17:28:13	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	440,640	668,380	1.5168
26 Dec 2011 12:25:52	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	414,720	627,937	1.5141
24 Dec 2011 18:10:05	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	388,800	588,484	1.5136
22 Dec 2011 19:15:37	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	362,880	549,557	1.5144
18 Dec 2011 12:53:20	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	336,960	510,692	1.5156
12 Dec 2011 18:53:28	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	311,040	471,103	1.5146
09 Dec 2011 18:40:50	1121257	13602546	hadcm3n_y9ue_1980_40_007536848_1	285,120	432,201	1.5159