Task 13410359

Name	hadcm3n_u5dr_1980_40_007459776_1
Workunit	7657279
Created	22 Sep 2011, 21:02:08 UTC
Sent	24 Sep 2011, 16:31:17 UTC
Report deadline	24 Dec 2011, 23:58:28 UTC
Received	5 Nov 2011, 15:48:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1287172
Run time	11 days 2 hours 3 min 55 sec
CPU time	9 days 6 hours 35 min 19 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 13:45:31 (5380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:44:02 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:26:09 (4676): No heartbeat from core client for 30 sec - exiting 04:26:10 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:03:51 (4504): No heartbeat from core client for 30 sec - exiting 07:03:52 (4504): No heartbeat from core client for 30 sec - exiting 07:03:54 (4504): No heartbeat from core client for 30 sec - exiting 07:03:55 (4504): No heartbeat from core client for 30 sec - exiting 07:03:56 (4504): No heartbeat from core client for 30 sec - exiting 07:03:57 (4504): No heartbeat from core client for 30 sec - exiting 07:03:58 (4504): No heartbeat from core client for 30 sec - exiting 07:03:59 (4504): No heartbeat from core client for 30 sec - exiting 07:04:00 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:06 (1836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:14:02 (4788): No heartbeat from core client for 30 sec - exiting 07:14:03 (4788): No heartbeat from core client for 30 sec - exiting 07:14:04 (4788): No heartbeat from core client for 30 sec - exiting 07:14:06 (4788): No heartbeat from core client for 30 sec - exiting 07:14:07 (4788): No heartbeat from core client for 30 sec - exiting 07:14:08 (4788): No heartbeat from core client for 30 sec - exiting 07:14:09 (4788): No heartbeat from core client for 30 sec - exiting 07:14:10 (4788): No heartbeat from core client for 30 sec - exiting 07:14:11 (4788): No heartbeat from core client for 30 sec - exiting 07:14:12 (4788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:36:45 (4740): No heartbeat from core client for 30 sec - exiting 05:36:46 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1512, selfPID=1512, iMonCtr=1 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u5drko.pji7c10 Error converting file to netcdf: dataout/u5drko.pii7c10 Error converting file to netcdf: dataout/u5drko.pfi7c10 Error converting file to netcdf: dataout/u5drka.phi7c10 Error converting file to netcdf: dataout/u5drka.pgi7c10 Error converting file to netcdf: dataout/u5drka.pei7c10 Error converting file to netcdf: dataout/u5drka.pdi7c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:37:39 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4304, selfPID=4304, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77DC8CFF read attempt to address 0x3E19209A Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1884, selfPID=1884, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77DF2AB1 read attempt to address 0x3F326E3B Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_u5dr_1980_40_007459776/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Nov 2011 13:13:10	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	518,400	798,863	1.5410
02 Nov 2011 12:22:49	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	492,480	758,951	1.5411
31 Oct 2011 18:17:47	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	466,560	719,170	1.5414
31 Oct 2011 15:46:22	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	440,640	681,448	1.5465
31 Oct 2011 14:05:05	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	414,720	641,768	1.5475
18 Oct 2011 17:13:38	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	388,800	602,198	1.5489
15 Oct 2011 20:30:26	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	362,880	562,815	1.5510
15 Oct 2011 06:10:00	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	336,960	523,245	1.5528
14 Oct 2011 05:31:26	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	311,040	483,153	1.5533
13 Oct 2011 13:20:12	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	285,120	450,340	1.5795
11 Oct 2011 23:42:10	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	259,200	410,938	1.5854
11 Oct 2011 10:56:14	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	233,280	371,637	1.5931
10 Oct 2011 21:01:43	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	207,360	331,822	1.6002
09 Oct 2011 10:33:16	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	181,440	291,274	1.6053
07 Oct 2011 21:42:10	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	155,520	250,429	1.6103
06 Oct 2011 10:30:00	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	129,600	209,619	1.6174
02 Oct 2011 11:58:21	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	103,680	168,291	1.6232
29 Sep 2011 02:02:28	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	77,760	126,548	1.6274
28 Sep 2011 01:03:40	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	51,840	84,448	1.6290
27 Sep 2011 07:22:12	988221	13410359	hadcm3n_u5dr_1980_40_007459776_1	25,920	42,243	1.6297