Task 13553023

Name	hadcm3n_ychj_1900_40_007523662_0
Workunit	7721137
Created	28 Oct 2011, 13:23:31 UTC
Sent	31 Oct 2011, 9:48:20 UTC
Report deadline	30 Jan 2012, 17:15:31 UTC
Received	3 Dec 2011, 11:08:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1146757
Run time	4 days 2 hours 47 min 2 sec
CPU time	4 days 1 hours 43 min 8 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:58:47 (3360): No heartbeat from core client for 30 sec - exiting 19:58:48 (3360): No heartbeat from core client for 30 sec - exiting 19:58:49 (3360): No heartbeat from core client for 30 sec - exiting 19:58:50 (3360): No heartbeat from core client for 30 sec - exiting 19:58:51 (3360): No heartbeat from core client for 30 sec - exiting 19:58:53 (3360): No heartbeat from core client for 30 sec - exiting 19:58:54 (3360): No heartbeat from core client for 30 sec - exiting 19:58:55 (3360): No heartbeat from core client for 30 sec - exiting 19:58:56 (3360): No heartbeat from core client for 30 sec - exiting 19:58:57 (3360): No heartbeat from core client for 30 sec - exiting 19:58:58 (3360): No heartbeat from core client for 30 sec - exiting 19:58:59 (3360): No heartbeat from core client for 30 sec - exiting 19:59:00 (3360): No heartbeat from core client for 30 sec - exiting 19:59:01 (3360): No heartbeat from core client for 30 sec - exiting 19:59:02 (3360): No heartbeat from core client for 30 sec - exiting 19:59:03 (3360): No heartbeat from core client for 30 sec - exiting 19:59:05 (3360): No heartbeat from core client for 30 sec - exiting 19:59:06 (3360): No heartbeat from core client for 30 sec - exiting 19:59:07 (3360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:04:09 (3684): No heartbeat from core client for 30 sec - exiting 16:04:10 (3684): No heartbeat from core client for 30 sec - exiting 16:04:11 (3684): No heartbeat from core client for 30 sec - exiting 16:04:12 (3684): No heartbeat from core client for 30 sec - exiting 16:04:13 (3684): No heartbeat from core client for 30 sec - exiting 16:04:14 (3684): No heartbeat from core client for 30 sec - exiting 16:04:15 (3684): No heartbeat from core client for 30 sec - exiting 16:04:16 (3684): No heartbeat from core client for 30 sec - exiting 16:04:18 (3684): No heartbeat from core client for 30 sec - exiting 16:04:19 (3684): No heartbeat from core client for 30 sec - exiting 16:04:20 (3684): No heartbeat from core client for 30 sec - exiting 16:04:21 (3684): No heartbeat from core client for 30 sec - exiting 16:04:22 (3684): No heartbeat from core client for 30 sec - exiting 16:04:23 (3684): No heartbeat from core client for 30 sec - exiting 16:04:24 (3684): No heartbeat from core client for 30 sec - exiting 16:04:25 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:03:34 (3688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=640, iMonCtr=1 Model crash detected, will try to restart... 21:56:19 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:18:17 (3856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2344, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:07:20 (4352): No heartbeat from core client for 30 sec - exiting 19:07:21 (4352): No heartbeat from core client for 30 sec - exiting 19:07:22 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:52:39 (3732): No heartbeat from core client for 30 sec - exiting 14:52:40 (3732): No heartbeat from core client for 30 sec - exiting 14:52:41 (3732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3160, iMonCtr=1 Model crash detected, will try to restart... 20:45:33 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:11:42 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:46:02 (3896): No heartbeat from core client for 30 sec - exiting 09:46:03 (3896): No heartbeat from core client for 30 sec - exiting 09:46:04 (3896): No heartbeat from core client for 30 sec - exiting 09:46:05 (3896): No heartbeat from core client for 30 sec - exiting 09:46:06 (3896): No heartbeat from core client for 30 sec - exiting 09:46:08 (3896): No heartbeat from core client for 30 sec - exiting 09:46:09 (3896): No heartbeat from core client for 30 sec - exiting 09:46:10 (3896): No heartbeat from core client for 30 sec - exiting 09:46:11 (3896): No heartbeat from core client for 30 sec - exiting 09:46:12 (3896): No heartbeat from core client for 30 sec - exiting 09:46:13 (3896): No heartbeat from core client for 30 sec - exiting 09:46:14 (3896): No heartbeat from core client for 30 sec - exiting 09:46:15 (3896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:46:03 (3816): No heartbeat from core client for 30 sec - exiting 19:46:04 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:05 (3816): No heartbeat from core client for 30 sec - exiting 19:46:06 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4672, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ychjko.pja9c10 Error converting file to netcdf: dataout/ychjko.pia9c10 Error converting file to netcdf: dataout/ychjko.pfa9c10 Error converting file to netcdf: dataout/ychjka.pha9c10 Error converting file to netcdf: dataout/ychjka.pga9c10 Error converting file to netcdf: dataout/ychjka.pea9c10 Error converting file to netcdf: dataout/ychjka.pda9c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ychjko.pja9c10 Error converting file to netcdf: dataout/ychjko.pia9c10 Error converting file to netcdf: dataout/ychjko.pfa9c10 Error converting file to netcdf: dataout/ychjka.pha9c10 Error converting file to netcdf: dataout/ychjka.pga9c10 Error converting file to netcdf: dataout/ychjka.pea9c10 Error converting file to netcdf: dataout/ychjka.pda9c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ychjko.pja9c10 Error converting file to netcdf: dataout/ychjko.pia9c10 Error converting file to netcdf: dataout/ychjko.pfa9c10 Error converting file to netcdf: dataout/ychjka.pha9c10 Error converting file to netcdf: dataout/ychjka.pga9c10 Error converting file to netcdf: dataout/ychjka.pea9c10 Error converting file to netcdf: dataout/ychjka.pda9c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ychjko.pja9c10 Error converting file to netcdf: dataout/ychjko.pia9c10 Error converting file to netcdf: dataout/ychjko.pfa9c10 Error converting file to netcdf: dataout/ychjka.pha9c10 Error converting file to netcdf: dataout/ychjka.pga9c10 Error converting file to netcdf: dataout/ychjka.pea9c10 Error converting file to netcdf: dataout/ychjka.pda9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77623A93 read attempt to address 0x40B9BC36 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\Program Files\BOINC\Data/projects/climateprediction.net/hadcm3n_ychj_1900_40_007523662/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Dec 2011 11:11:48	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	259,200	351,787	1.3572
27 Nov 2011 21:21:29	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	233,280	316,256	1.3557
26 Nov 2011 09:59:53	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	207,360	281,044	1.3553
21 Nov 2011 17:20:47	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	181,440	245,556	1.3534
15 Nov 2011 16:47:16	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	155,520	210,626	1.3543
15 Nov 2011 16:47:16	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	129,600	175,941	1.3576
15 Nov 2011 16:47:16	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	103,680	140,904	1.3590
05 Nov 2011 16:11:01	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	77,760	105,678	1.3590
01 Nov 2011 19:01:59	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	51,840	70,474	1.3595
01 Nov 2011 08:12:31	1146757	13553023	hadcm3n_ychj_1900_40_007523662_0	25,920	35,160	1.3565