Task 13641388

Name	hadcm3n_t42u_1940_40_007536320_2
Workunit	7733552
Created	17 Nov 2011, 7:04:30 UTC
Sent	17 Nov 2011, 7:07:35 UTC
Report deadline	16 Feb 2012, 14:34:46 UTC
Received	17 Feb 2012, 16:11:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1145695
Run time	32 days 13 hours 17 min 48 sec
CPU time	31 days 2 hours 51 min 47 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5416, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t42uko.pje5c10 Error converting file to netcdf: dataout/t42uko.pie5c10 Error converting file to netcdf: dataout/t42uko.pfe5c10 Error converting file to netcdf: dataout/t42uka.phe5c10 Error converting file to netcdf: dataout/t42uka.pge5c10 Error converting file to netcdf: dataout/t42uka.pee5c10 Error converting file to netcdf: dataout/t42uka.pde5c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5420, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5612, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5480, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5156, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t42uko.pjf6c10 Error converting file to netcdf: dataout/t42uko.pif6c10 Error converting file to netcdf: dataout/t42uko.pff6c10 Error converting file to netcdf: dataout/t42uka.phf6c10 Error converting file to netcdf: dataout/t42uka.pgf6c10 Error converting file to netcdf: dataout/t42uka.pef6c10 Error converting file to netcdf: dataout/t42uka.pdf6c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3076, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4920, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4304, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5484, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5276, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x776D5EAB read attempt to address 0x404BA84C Engaging BOINC Windows Runtime Debugger... Cannot serialize file D:\BOINC DATA/projects/climateprediction.net/hadcm3n_t42u_1940_40_007536320/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Feb 2012 16:13:50	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	1,036,800	2,688,701	2.5933
15 Feb 2012 12:00:07	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	1,010,880	2,618,602	2.5904
13 Feb 2012 15:16:49	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	984,960	2,549,516	2.5884
10 Feb 2012 13:23:14	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	959,040	2,484,336	2.5904
08 Feb 2012 10:45:16	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	933,120	2,415,937	2.5891
06 Feb 2012 10:46:37	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	907,200	2,347,821	2.5880
02 Feb 2012 19:52:16	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	881,280	2,282,213	2.5897
01 Feb 2012 07:28:24	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	855,360	2,214,366	2.5888
29 Jan 2012 17:06:47	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	829,440	2,147,342	2.5889
26 Jan 2012 08:11:21	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	803,520	2,081,324	2.5903
23 Jan 2012 16:32:26	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	777,600	2,012,801	2.5885
21 Jan 2012 14:52:06	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	751,680	1,944,638	2.5871
18 Jan 2012 08:59:12	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	725,760	1,876,858	2.5861
16 Jan 2012 10:55:11	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	699,840	1,809,715	2.5859
14 Jan 2012 15:58:42	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	673,920	1,743,288	2.5868
12 Jan 2012 11:54:04	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	648,000	1,675,265	2.5853
09 Jan 2012 19:54:40	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	622,080	1,606,874	2.5831
07 Jan 2012 16:59:00	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	596,160	1,540,177	2.5835
05 Jan 2012 20:04:16	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	570,240	1,472,256	2.5818
04 Jan 2012 14:52:36	1145695	13641388	hadcm3n_t42u_1940_40_007536320_2	544,320	1,404,374	2.5801