Task 13812787

Name	hadcm3n_yk6n_1940_40_007617586_2
Workunit	7795716
Created	23 Dec 2011, 12:07:04 UTC
Sent	23 Dec 2011, 12:26:15 UTC
Report deadline	23 Mar 2012, 19:53:26 UTC
Received	28 Mar 2012, 4:35:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	911595
Run time	37 days 15 hours 19 min 11 sec
CPU time	31 days 8 hours 51 min 33 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	1.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:02:34 (1080): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day 19:22:50 (3900): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day forrtl: The file or directory is corrupted and unreadable. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3620, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day forrtl: The file or directory is corrupted and unreadable. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3620, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day forrtl: The file or directory is corrupted and unreadable. 19:58:44 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day forrtl: The file or directory is corrupted and unreadable. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2776, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day forrtl: The file or directory is corrupted and unreadable. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2776, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error reading file E:\Application Data\BOINC/projects/climateprediction.net/hadcm3n_yk6n_1940_40_007617586/dataout/ocean_restart.day forrtl: The file or directory is corrupted and unreadable. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2776, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:49:09 (2960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:10 (2960): No heartbeat from core client for 30 sec - exiting 22:49:11 (2960): No heartbeat from core client for 30 sec - exiting 22:49:12 (2960): No heartbeat from core client for 30 sec - exiting 22:49:13 (2960): No heartbeat from core client for 30 sec - exiting 22:49:15 (2960): No heartbeat from core client for 30 sec - exiting 12:19:07 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:14:18 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:24:33 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:11 (2856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3648, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Mar 2012 17:10:49	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	440,640	2,667,042	6.0527
12 Mar 2012 15:56:14	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	414,720	2,513,663	6.0611
10 Mar 2012 14:29:10	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	388,800	2,356,081	6.0599
08 Mar 2012 08:45:26	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	362,880	2,203,421	6.0720
05 Mar 2012 23:30:37	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	336,960	2,049,862	6.0834
03 Mar 2012 13:35:00	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	311,040	1,896,305	6.0967
01 Mar 2012 08:10:21	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	285,120	1,744,640	6.1190
07 Feb 2012 13:11:00	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	259,200	1,588,917	6.1301
05 Feb 2012 14:50:52	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	233,280	1,437,887	6.1638
03 Feb 2012 13:59:33	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	207,360	1,283,906	6.1917
01 Feb 2012 15:07:08	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	181,440	1,131,506	6.2363
30 Jan 2012 11:26:07	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	155,520	972,005	6.2500
28 Jan 2012 03:02:25	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	129,600	819,596	6.3240
25 Jan 2012 21:40:37	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	103,680	659,364	6.3596
23 Jan 2012 19:23:35	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	77,760	496,251	6.3818
21 Jan 2012 09:20:52	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	51,840	328,626	6.3392
14 Jan 2012 03:05:57	911595	13812787	hadcm3n_yk6n_1940_40_007617586_2	25,920	171,361	6.6111