Task 15655748

Name	hadcm3n_zi1x_1920_40_008320698_3
Workunit	8471833
Created	10 Mar 2013, 8:24:10 UTC
Sent	10 Mar 2013, 8:24:16 UTC
Report deadline	9 Jun 2013, 15:51:27 UTC
Received	20 Mar 2013, 11:14:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	471651
Run time	8 days 19 hours 40 min 47 sec
CPU time	8 days 19 hours 40 min 47 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	1.23 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.10.45</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 21:10:58 (9620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:03 (9620): No heartbeat from core client for 30 sec - exiting 21:11:04 (9620): No heartbeat from core client for 30 sec - exiting 21:11:08 (9620): No heartbeat from core client for 30 sec - exiting 21:11:09 (9620): No heartbeat from core client for 30 sec - exiting 21:12:05 (8412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:24:32 (9592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:25:54 (6496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:21 (11964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:22 (11964): No heartbeat from core client for 30 sec - exiting 05:54:23 (11964): No heartbeat from core client for 30 sec - exiting 05:54:24 (11964): No heartbeat from core client for 30 sec - exiting 05:54:25 (11964): No heartbeat from core client for 30 sec - exiting 11:40:45 (14092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:41:52 (8096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:06 (10804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:07 (10804): No heartbeat from core client for 30 sec - exiting 22:27:08 (10804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:00:50 (12132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:54 (12132): No heartbeat from core client for 30 sec - exiting 23:07:22 (7008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:07:24 (7008): No heartbeat from core client for 30 sec - exiting 23:09:33 (14268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:34 (9172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:36 (9172): No heartbeat from core client for 30 sec - exiting 23:14:37 (9172): No heartbeat from core client for 30 sec - exiting 23:14:38 (9172): No heartbeat from core client for 30 sec - exiting 23:14:39 (9172): No heartbeat from core client for 30 sec - exiting 23:16:04 (15712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:22:12 (14472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:24:37 (15044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:04 (14536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:05 (14536): No heartbeat from core client for 30 sec - exiting 23:37:23 (17312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:24 (17312): No heartbeat from core client for 30 sec - exiting 23:39:39 (16768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:41 (18060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:42 (18060): No heartbeat from core client for 30 sec - exiting 23:44:43 (18060): No heartbeat from core client for 30 sec - exiting 23:44:44 (18060): No heartbeat from core client for 30 sec - exiting 23:44:45 (18060): No heartbeat from core client for 30 sec - exiting 23:45:47 (11852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:51:51 (17636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:43 (17792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:51 (17296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:52 (17296): No heartbeat from core client for 30 sec - exiting 00:07:27 (15164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:07:28 (15164): No heartbeat from core client for 30 sec - exiting 00:09:46 (15312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:47 (14372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:27 (17656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:19:48 (16236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:19:49 (16236): No heartbeat from core client for 30 sec - exiting 00:20:39 (17432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:21:36 (17592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:07 (15164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:29:50 (10004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:50 (17952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:51 (17952): No heartbeat from core client for 30 sec - exiting 00:34:52 (17952): No heartbeat from core client for 30 sec - exiting 00:35:36 (18208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:39 (17932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:40 (17932): No heartbeat from core client for 30 sec - exiting 00:49:53 (17268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:55:29 (16900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:55:34 (16900): No heartbeat from core client for 30 sec - exiting 00:55:35 (16900): No heartbeat from core client for 30 sec - exiting 00:55:36 (16900): No heartbeat from core client for 30 sec - exiting 00:55:37 (16900): No heartbeat from core client for 30 sec - exiting 00:55:38 (16900): No heartbeat from core client for 30 sec - exiting 00:55:39 (16900): No heartbeat from core client for 30 sec - exiting 00:55:40 (16900): No heartbeat from core client for 30 sec - exiting 00:55:41 (16900): No heartbeat from core client for 30 sec - exiting 00:56:58 (16252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:04 (17772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:06 (17772): No heartbeat from core client for 30 sec - exiting 01:05:31 (16916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:32 (18228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:33 (18228): No heartbeat from core client for 30 sec - exiting 01:10:34 (18228): No heartbeat from core client for 30 sec - exiting 01:10:35 (18228): No heartbeat from core client for 30 sec - exiting 01:11:24 (17608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:01 (16960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:02 (16960): No heartbeat from core client for 30 sec - exiting 01:20:32 (17508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:21:19 (10348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:35 (17952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:36 (17952): No heartbeat from core client for 30 sec - exiting 01:26:08 (17952): No heartbeat from core client for 30 sec - exiting 01:26:09 (17952): No heartbeat from core client for 30 sec - exiting 01:30:34 (14176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:30:36 (14176): No heartbeat from core client for 30 sec - exiting 01:31:32 (18216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:55 (17348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:56 (17348): No heartbeat from core client for 30 sec - exiting 01:40:36 (18020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:39 (17592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:40 (17592): No heartbeat from core client for 30 sec - exiting 01:46:43 (17588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:04 (17580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:05 (17580): No heartbeat from core client for 30 sec - exiting 01:55:40 (15716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:03 (18300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:00:49 (17556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:21 (16688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:22 (16688): No heartbeat from core client for 30 sec - exiting 23:11:23 (16688): No heartbeat from core client for 30 sec - exiting 23:11:24 (16688): No heartbeat from core client for 30 sec - exiting 23:17:12 (20256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:13 (20256): No heartbeat from core client for 30 sec - exiting 23:20:14 (19768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:17 (20056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:18 (20056): No heartbeat from core client for 30 sec - exiting 23:25:19 (20056): No heartbeat from core client for 30 sec - exiting 23:26:11 (20032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:35:18 (20408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:19 (19052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:22 (19300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:37 (20184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:03:30 (16768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zi1xko.pjc5c10 Error converting file to netcdf: dataout/zi1xko.pic5c10 Error converting file to netcdf: dataout/zi1xko.pfc5c10 Error converting file to netcdf: dataout/zi1xka.phc5c10 Error converting file to netcdf: dataout/zi1xka.pgc5c10 Error converting file to netcdf: dataout/zi1xka.pec5c10 Error converting file to netcdf: dataout/zi1xka.pdc5c10 00:05:13 (18896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 08:03:47 (21988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:49 (21988): No heartbeat from core client for 30 sec - exiting 08:04:32 (19076): No heartbeat from core client for 30 sec - exiting 08:04:33 (19076): No heartbeat from core client for 30 sec - exiting 08:04:34 (19076): No heartbeat from core client for 30 sec - exiting 08:04:35 (19076): No heartbeat from core client for 30 sec - exiting 08:04:37 (19076): No heartbeat from core client for 30 sec - exiting 08:04:38 (19076): No heartbeat from core client for 30 sec - exiting 08:04:39 (19076): No heartbeat from core client for 30 sec - exiting 08:04:40 (19076): No heartbeat from core client for 30 sec - exiting 08:04:41 (19076): No heartbeat from core client for 30 sec - exiting 08:04:42 (19076): No heartbeat from core client for 30 sec - exiting 08:04:43 (19076): No heartbeat from core client for 30 sec - exiting 08:04:44 (19076): No heartbeat from core client for 30 sec - exiting 08:04:45 (19076): No heartbeat from core client for 30 sec - exiting 08:04:46 (19076): No heartbeat from core client for 30 sec - exiting 08:04:47 (19076): No heartbeat from core client for 30 sec - exiting 08:04:48 (19076): No heartbeat from core client for 30 sec - exiting 08:04:49 (19076): No heartbeat from core client for 30 sec - exiting 08:04:50 (19076): No heartbeat from core client for 30 sec - exiting 08:04:51 (19076): No heartbeat from core client for 30 sec - exiting 08:04:52 (19076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:38 (22016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:45 (22016): No heartbeat from core client for 30 sec - exiting 08:05:46 (22016): No heartbeat from core client for 30 sec - exiting 08:05:47 (22016): No heartbeat from core client for 30 sec - exiting 08:05:48 (22016): No heartbeat from core client for 30 sec - exiting 08:05:49 (22016): No heartbeat from core client for 30 sec - exiting 08:05:50 (22016): No heartbeat from core client for 30 sec - exiting 08:05:51 (22016): No heartbeat from core client for 30 sec - exiting 08:05:52 (22016): No heartbeat from core client for 30 sec - exiting 08:05:53 (22016): No heartbeat from core client for 30 sec - exiting 08:06:53 (22300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:44 (20432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Mar 2013 04:28:02	471651	15655748	hadcm3n_zi1x_1920_40_008320698_3	129,600	680,658	5.2520
17 Mar 2013 10:29:23	471651	15655748	hadcm3n_zi1x_1920_40_008320698_3	103,680	546,963	5.2755
15 Mar 2013 13:50:13	471651	15655748	hadcm3n_zi1x_1920_40_008320698_3	77,760	407,328	5.2383
13 Mar 2013 18:59:08	471651	15655748	hadcm3n_zi1x_1920_40_008320698_3	51,840	269,026	5.1895
12 Mar 2013 01:24:23	471651	15655748	hadcm3n_zi1x_1920_40_008320698_3	25,920	133,702	5.1583