Task 13596821

Name	hadcm3n_o107_1980_40_007534722_0
Workunit	7731954
Created	5 Nov 2011, 5:09:57 UTC
Sent	5 Nov 2011, 5:12:12 UTC
Report deadline	4 Feb 2012, 12:39:23 UTC
Received	28 Nov 2011, 19:52:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1172319
Run time	8 days 0 hours 39 min 55 sec
CPU time	7 days 17 hours 20 min 11 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:15:06 (5652): No heartbeat from core client for 30 sec - exiting 10:15:07 (5652): No heartbeat from core client for 30 sec - exiting 10:15:08 (5652): No heartbeat from core client for 30 sec - exiting 10:15:09 (5652): No heartbeat from core client for 30 sec - exiting 10:15:10 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:09:36 (3244): No heartbeat from core client for 30 sec - exiting 11:09:37 (3244): No heartbeat from core client for 30 sec - exiting 11:09:38 (3244): No heartbeat from core client for 30 sec - exiting 11:09:39 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:54:21 (2896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o107ko.pjj8c10 Error converting file to netcdf: dataout/o107ko.pij8c10 Error converting file to netcdf: dataout/o107ko.pfj8c10 Error converting file to netcdf: dataout/o107ka.phj8c10 Error converting file to netcdf: dataout/o107ka.pgj8c10 Error converting file to netcdf: dataout/o107ka.pej8c10 Error converting file to netcdf: dataout/o107ka.pdj8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Nov 2011 02:53:49	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	466,560	637,194	1.3657
27 Nov 2011 13:06:51	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	440,640	602,216	1.3667
27 Nov 2011 01:18:29	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	414,720	567,303	1.3679
26 Nov 2011 14:36:00	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	388,800	533,345	1.3718
26 Nov 2011 01:57:52	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	362,880	498,646	1.3741
25 Nov 2011 13:37:11	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	336,960	463,677	1.3761
24 Nov 2011 10:37:03	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	311,040	428,748	1.3784
21 Nov 2011 02:16:55	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	285,120	392,075	1.3751
20 Nov 2011 00:20:54	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	259,200	355,727	1.3724
18 Nov 2011 15:41:03	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	233,280	320,190	1.3726
17 Nov 2011 14:29:50	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	207,360	284,641	1.3727
16 Nov 2011 15:52:22	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	181,440	248,591	1.3701
15 Nov 2011 18:44:16	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	155,520	212,753	1.3680
15 Nov 2011 18:44:16	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	129,600	177,477	1.3694
15 Nov 2011 18:44:16	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	103,680	142,463	1.3741
15 Nov 2011 18:44:16	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	77,760	107,164	1.3781
15 Nov 2011 18:44:16	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	51,840	71,561	1.3804
15 Nov 2011 18:44:16	1172319	13596821	hadcm3n_o107_1980_40_007534722_0	25,920	35,078	1.3533