Task 15812510

Name	hadcm3n_o3lx_1940_40_008381268_0
Workunit	8532127
Created	1 Jun 2013, 0:46:45 UTC
Sent	15 Jun 2013, 5:23:00 UTC
Report deadline	14 Sep 2013, 12:50:11 UTC
Received	29 Jun 2013, 21:33:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1377383
Run time	6 days 14 hours 1 min 52 sec
CPU time	4 days 4 hours 1 min 39 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 05:40:43 (436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:41 (7160): No heartbeat from core client for 30 sec - exiting 08:45:42 (7160): No heartbeat from core client for 30 sec - exiting 08:45:43 (7160): No heartbeat from core client for 30 sec - exiting 08:45:44 (7160): No heartbeat from core client for 30 sec - exiting 08:45:45 (7160): No heartbeat from core client for 30 sec - exiting 08:45:46 (7160): No heartbeat from core client for 30 sec - exiting 08:45:47 (7160): No heartbeat from core client for 30 sec - exiting 08:45:48 (7160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:27:49 (11020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:06:14 (9544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:38:44 (8636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:28:45 (6656): No heartbeat from core client for 30 sec - exiting 03:28:46 (6656): No heartbeat from core client for 30 sec - exiting 03:28:47 (6656): No heartbeat from core client for 30 sec - exiting 03:28:48 (6656): No heartbeat from core client for 30 sec - exiting 03:28:49 (6656): No heartbeat from core client for 30 sec - exiting 03:28:50 (6656): No heartbeat from core client for 30 sec - exiting 03:28:51 (6656): No heartbeat from core client for 30 sec - exiting 03:28:52 (6656): No heartbeat from core client for 30 sec - exiting 03:28:53 (6656): No heartbeat from core client for 30 sec - exiting 03:28:55 (6656): No heartbeat from core client for 30 sec - exiting 03:28:56 (6656): No heartbeat from core client for 30 sec - exiting 03:28:57 (6656): No heartbeat from core client for 30 sec - exiting 03:28:58 (6656): No heartbeat from core client for 30 sec - exiting 03:28:59 (6656): No heartbeat from core client for 30 sec - exiting 03:29:01 (6656): No heartbeat from core client for 30 sec - exiting 03:29:02 (6656): No heartbeat from core client for 30 sec - exiting 03:29:03 (6656): No heartbeat from core client for 30 sec - exiting 03:29:05 (6656): No heartbeat from core client for 30 sec - exiting 03:29:06 (6656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:29:07 (6656): No heartbeat from core client for 30 sec - exiting 03:29:08 (6656): No heartbeat from core client for 30 sec - exiting 03:29:09 (6656): No heartbeat from core client for 30 sec - exiting 03:29:10 (6656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o3lxko.pje3c10 Error converting file to netcdf: dataout/o3lxko.pie3c10 Error converting file to netcdf: dataout/o3lxko.pfe3c10 Error converting file to netcdf: dataout/o3lxka.phe3c10 Error converting file to netcdf: dataout/o3lxka.pge3c10 Error converting file to netcdf: dataout/o3lxka.pee3c10 Error converting file to netcdf: dataout/o3lxka.pde3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:26:31 (10824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:44 (7676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3596, selfPID=3596, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9944, selfPID=9944, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o3lxko.pje6c10 Error converting file to netcdf: dataout/o3lxko.pie6c10 Error converting file to netcdf: dataout/o3lxko.pfe6c10 Error converting file to netcdf: dataout/o3lxka.phe6c10 Error converting file to netcdf: dataout/o3lxka.pge6c10 Error converting file to netcdf: dataout/o3lxka.pee6c10 Ocean Restart file copy failed on o3lxko.dae73d0 Suspended CPDN Monitor - Suspend request from BOINC... 12:58:43 (1288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3lxko.dae81s0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Jun 2013 06:57:41	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	181,440	329,414	1.8156
27 Jun 2013 06:12:52	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	155,520	280,638	1.8045
25 Jun 2013 16:15:56	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	129,600	231,469	1.7860
21 Jun 2013 12:30:16	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	103,680	184,930	1.7837
18 Jun 2013 22:15:57	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	77,760	140,009	1.8005
18 Jun 2013 04:06:40	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	51,840	93,955	1.8124
16 Jun 2013 19:31:56	859807	15812510	hadcm3n_o3lx_1940_40_008381268_0	25,920	47,477	1.8317