Task 15636568

Name	hadcm3n_n2dc_1920_40_008321384_0
Workunit	8472519
Created	24 Feb 2013, 19:13:01 UTC
Sent	24 Feb 2013, 19:14:07 UTC
Report deadline	27 May 2013, 2:41:18 UTC
Received	6 Mar 2013, 3:30:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1244094
Run time	7 days 5 hours 48 min 45 sec
CPU time	7 days 0 hours 25 min 21 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:52:51 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:23:23 (9192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:24 (9192): No heartbeat from core client for 30 sec - exiting 06:38:25 (8496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:38:27 (8496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n2dcko.pjc4c10 Error converting file to netcdf: dataout/n2dcko.pic4c10 Error converting file to netcdf: dataout/n2dcko.pfc4c10 Error converting file to netcdf: dataout/n2dcka.phc4c10 Error converting file to netcdf: dataout/n2dcka.pgc4c10 Error converting file to netcdf: dataout/n2dcka.pec4c10 Error converting file to netcdf: dataout/n2dcka.pdc4c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n2dcko.pjc4c10 Error converting file to netcdf: dataout/n2dcko.pic4c10 Error converting file to netcdf: dataout/n2dcko.pfc4c10 Error converting file to netcdf: dataout/n2dcka.phc4c10 Error converting file to netcdf: dataout/n2dcka.pgc4c10 Error converting file to netcdf: dataout/n2dcka.pec4c10 Error converting file to netcdf: dataout/n2dcka.pdc4c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n2dcko.pjc4c10 Error converting file to netcdf: dataout/n2dcko.pic4c10 Error converting file to netcdf: dataout/n2dcko.pfc4c10 Error converting file to netcdf: dataout/n2dcka.phc4c10 Error converting file to netcdf: dataout/n2dcka.pgc4c10 Error converting file to netcdf: dataout/n2dcka.pec4c10 Error converting file to netcdf: dataout/n2dcka.pdc4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:12:25 (8604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:27 (8604): No heartbeat from core client for 30 sec - exiting 10:12:30 (8604): No heartbeat from core client for 30 sec - exiting 10:12:32 (8604): No heartbeat from core client for 30 sec - exiting 10:12:35 (8604): No heartbeat from core client for 30 sec - exiting 10:12:37 (8604): No heartbeat from core client for 30 sec - exiting 10:17:31 (2604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:33 (2604): No heartbeat from core client for 30 sec - exiting 10:17:35 (2604): No heartbeat from core client for 30 sec - exiting 10:23:24 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:23:26 (5196): No heartbeat from core client for 30 sec - exiting 10:23:31 (5196): No heartbeat from core client for 30 sec - exiting 10:23:32 (5196): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:29:19 (3236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:18 (4416): No heartbeat from core client for 30 sec - exiting 10:31:20 (4416): No heartbeat from core client for 30 sec - exiting 10:31:23 (4416): No heartbeat from core client for 30 sec - exiting 10:31:25 (4416): No heartbeat from core client for 30 sec - exiting 10:31:28 (4416): No heartbeat from core client for 30 sec - exiting 10:31:30 (4416): No heartbeat from core client for 30 sec - exiting 10:31:32 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:35 (4416): No heartbeat from core client for 30 sec - exiting 10:31:37 (4416): No heartbeat from core client for 30 sec - exiting 10:31:40 (4416): No heartbeat from core client for 30 sec - exiting 10:31:42 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 02:42:01 (7848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:22:23 (5104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:22:27 (5104): No heartbeat from core client for 30 sec - exiting 14:22:29 (5104): No heartbeat from core client for 30 sec - exiting 14:22:33 (5104): No heartbeat from core client for 30 sec - exiting 14:22:35 (5104): No heartbeat from core client for 30 sec - exiting 14:25:08 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Mar 2013 11:31:36	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	336,960	595,018	1.7658
04 Mar 2013 21:31:56	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	311,040	549,223	1.7658
04 Mar 2013 03:06:54	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	285,120	503,186	1.7648
03 Mar 2013 14:05:19	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	259,200	456,636	1.7617
03 Mar 2013 00:43:55	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	233,280	410,323	1.7589
02 Mar 2013 11:14:14	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	207,360	364,344	1.7571
01 Mar 2013 21:53:15	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	181,440	318,233	1.7539
01 Mar 2013 08:57:31	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	155,520	272,953	1.7551
28 Feb 2013 11:13:19	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	129,600	227,654	1.7566
27 Feb 2013 00:34:50	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	77,760	136,803	1.7593
26 Feb 2013 03:48:42	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	51,840	91,043	1.7562
25 Feb 2013 08:14:34	1244094	15636568	hadcm3n_n2dc_1920_40_008321384_0	25,920	45,767	1.7657