Task 14367426

Name	hadcm3n_yezw_1980_40_007858717_0
Workunit	8013829
Created	5 Apr 2012, 20:32:38 UTC
Sent	5 Apr 2012, 20:38:01 UTC
Report deadline	6 Jul 2012, 4:05:12 UTC
Received	8 Apr 2012, 12:40:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1107891
Run time	16 hours 8 min 57 sec
CPU time	13 hours 27 min 1 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.43 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 09:38:57 (2088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:39:01 (2088): No heartbeat from core client for 30 sec - exiting 09:39:02 (2088): No heartbeat from core client for 30 sec - exiting 09:39:03 (2088): No heartbeat from core client for 30 sec - exiting 09:39:05 (2088): No heartbeat from core client for 30 sec - exiting 09:39:06 (2088): No heartbeat from core client for 30 sec - exiting 09:39:07 (2088): No heartbeat from core client for 30 sec - exiting 09:39:08 (2088): No heartbeat from core client for 30 sec - exiting 09:41:38 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:05 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:07 (3624): No heartbeat from core client for 30 sec - exiting 10:39:08 (3624): No heartbeat from core client for 30 sec - exiting 10:39:09 (3624): No heartbeat from core client for 30 sec - exiting 10:39:10 (3624): No heartbeat from core client for 30 sec - exiting 10:39:12 (3624): No heartbeat from core client for 30 sec - exiting 10:39:13 (3624): No heartbeat from core client for 30 sec - exiting 11:39:11 (5548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:13 (5548): No heartbeat from core client for 30 sec - exiting 11:39:14 (5548): No heartbeat from core client for 30 sec - exiting 11:39:15 (5548): No heartbeat from core client for 30 sec - exiting 11:39:16 (5548): No heartbeat from core client for 30 sec - exiting 11:39:17 (5548): No heartbeat from core client for 30 sec - exiting 11:39:18 (5548): No heartbeat from core client for 30 sec - exiting 11:39:19 (5548): No heartbeat from core client for 30 sec - exiting 11:39:20 (5548): No heartbeat from core client for 30 sec - exiting 11:39:21 (5548): No heartbeat from core client for 30 sec - exiting 11:39:22 (5548): No heartbeat from core client for 30 sec - exiting 11:39:24 (5548): No heartbeat from core client for 30 sec - exiting 11:39:25 (5548): No heartbeat from core client for 30 sec - exiting 11:39:26 (5548): No heartbeat from core client for 30 sec - exiting 11:39:27 (5548): No heartbeat from core client for 30 sec - exiting 11:39:28 (5548): No heartbeat from core client for 30 sec - exiting 12:30:47 (2772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:23 (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:28 (4292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:56 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:56 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:59 (5468): No heartbeat from core client for 30 sec - exiting 13:40:00 (5468): No heartbeat from core client for 30 sec - exiting 13:40:01 (5468): No heartbeat from core client for 30 sec - exiting 13:40:02 (5468): No heartbeat from core client for 30 sec - exiting 14:06:11 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:07 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:11 (5056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:13 (5056): No heartbeat from core client for 30 sec - exiting 15:09:14 (5056): No heartbeat from core client for 30 sec - exiting 15:09:15 (5056): No heartbeat from core client for 30 sec - exiting 15:09:16 (5056): No heartbeat from core client for 30 sec - exiting 15:34:40 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:41 (6108): No heartbeat from core client for 30 sec - exiting 15:36:25 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:36:10 (788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:36:13 (788): No heartbeat from core client for 30 sec - exiting 16:36:14 (788): No heartbeat from core client for 30 sec - exiting 16:36:15 (788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:17:47 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:57 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:58 (3812): No heartbeat from core client for 30 sec - exiting 23:41:59 (3812): No heartbeat from core client for 30 sec - exiting 23:43:24 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:26 (3816): No heartbeat from core client for 30 sec - exiting 23:51:46 (5272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:51:50 (5272): No heartbeat from core client for 30 sec - exiting 23:51:51 (5272): No heartbeat from core client for 30 sec - exiting 23:51:52 (5272): No heartbeat from core client for 30 sec - exiting 23:51:53 (5272): No heartbeat from core client for 30 sec - exiting 23:51:54 (5272): No heartbeat from core client for 30 sec - exiting 23:51:55 (5272): No heartbeat from core client for 30 sec - exiting 23:51:57 (5272): No heartbeat from core client for 30 sec - exiting 23:51:58 (5272): No heartbeat from core client for 30 sec - exiting 23:51:59 (5272): No heartbeat from core client for 30 sec - exiting 23:52:00 (5272): No heartbeat from core client for 30 sec - exiting 00:39:40 (5220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:39:44 (5220): No heartbeat from core client for 30 sec - exiting 00:39:45 (5220): No heartbeat from core client for 30 sec - exiting 00:39:46 (5220): No heartbeat from core client for 30 sec - exiting 00:39:47 (5220): No heartbeat from core client for 30 sec - exiting 00:39:48 (5220): No heartbeat from core client for 30 sec - exiting 00:39:49 (5220): No heartbeat from core client for 30 sec - exiting 00:39:50 (5220): No heartbeat from core client for 30 sec - exiting 00:39:52 (5220): No heartbeat from core client for 30 sec - exiting 00:39:53 (5220): No heartbeat from core client for 30 sec - exiting 01:41:01 (2040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:41:02 (2040): No heartbeat from core client for 30 sec - exiting 01:41:45 (5176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:50 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:55 (5476): No heartbeat from core client for 30 sec - exiting 02:14:56 (5476): No heartbeat from core client for 30 sec - exiting 02:14:57 (5476): No heartbeat from core client for 30 sec - exiting 02:32:42 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:32:44 (5172): No heartbeat from core client for 30 sec - exiting 02:32:45 (5172): No heartbeat from core client for 30 sec - exiting 02:32:47 (5172): No heartbeat from core client for 30 sec - exiting 02:32:48 (5172): No heartbeat from core client for 30 sec - exiting 02:39:09 (4272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:11 (4272): No heartbeat from core client for 30 sec - exiting 02:39:12 (4272): No heartbeat from core client for 30 sec - exiting 02:39:13 (4272): No heartbeat from core client for 30 sec - exiting 02:39:14 (4272): No heartbeat from core client for 30 sec - exiting 02:39:15 (4272): No heartbeat from core client for 30 sec - exiting 02:39:17 (4272): No heartbeat from core client for 30 sec - exiting 02:39:18 (4272): No heartbeat from core client for 30 sec - exiting 02:39:19 (4272): No heartbeat from core client for 30 sec - exiting 02:45:54 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:57 (3104): No heartbeat from core client for 30 sec - exiting 02:45:59 (3104): No heartbeat from core client for 30 sec - exiting 02:46:00 (3104): No heartbeat from core client for 30 sec - exiting 02:46:01 (3104): No heartbeat from core client for 30 sec - exiting 02:46:02 (3104): No heartbeat from core client for 30 sec - exiting 02:46:03 (3104): No heartbeat from core client for 30 sec - exiting 02:46:04 (3104): No heartbeat from core client for 30 sec - exiting 02:47:46 (1544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:51 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:55 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:57 (5240): No heartbeat from core client for 30 sec - exiting 03:17:19 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:00 (5152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:02 (5152): No heartbeat from core client for 30 sec - exiting 03:20:03 (5152): No heartbeat from core client for 30 sec - exiting 03:22:13 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:23:20 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:32 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:37 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:24 (4012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:26 (4012): No heartbeat from core client for 30 sec - exiting 03:41:27 (4012): No heartbeat from core client for 30 sec - exiting 03:41:28 (4012): No heartbeat from core client for 30 sec - exiting 03:45:28 (5104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:30 (5104): No heartbeat from core client for 30 sec - exiting 03:45:31 (5104): No heartbeat from core client for 30 sec - exiting 03:45:32 (5104): No heartbeat from core client for 30 sec - exiting 04:01:36 (4076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:01:38 (4076): No heartbeat from core client for 30 sec - exiting 04:36:40 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:45 (5900): No heartbeat from core client for 30 sec - exiting 04:36:46 (5900): No heartbeat from core client for 30 sec - exiting 04:36:47 (5900): No heartbeat from core client for 30 sec - exiting 04:36:48 (5900): No heartbeat from core client for 30 sec - exiting 04:47:22 (2752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:47:24 (2752): No heartbeat from core client for 30 sec - exiting 04:47:25 (2752): No heartbeat from core client for 30 sec - exiting 04:57:25 (1544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:27 (1544): No heartbeat from core client for 30 sec - exiting 04:57:29 (1544): No heartbeat from core client for 30 sec - exiting 04:57:30 (1544): No heartbeat from core client for 30 sec - exiting 05:01:00 (5420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:01:02 (5420): No heartbeat from core client for 30 sec - exiting 05:02:43 (1948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:12:47 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:12:48 (4132): No heartbeat from core client for 30 sec - exiting 05:12:49 (4132): No heartbeat from core client for 30 sec - exiting 05:12:50 (4132): No heartbeat from core client for 30 sec - exiting 05:12:52 (4132): No heartbeat from core client for 30 sec - exiting 05:12:53 (4132): No heartbeat from core client for 30 sec - exiting 05:12:54 (4132): No heartbeat from core client for 30 sec - exiting 05:12:55 (4132): No heartbeat from core client for 30 sec - exiting 05:12:56 (4132): No heartbeat from core client for 30 sec - exiting 05:12:57 (4132): No heartbeat from core client for 30 sec - exiting 05:12:58 (4132): No heartbeat from core client for 30 sec - exiting 05:12:59 (4132): No heartbeat from core client for 30 sec - exiting 05:22:45 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:46 (5296): No heartbeat from core client for 30 sec - exiting 05:22:47 (5296): No heartbeat from core client for 30 sec - exiting 05:22:48 (5296): No heartbeat from core client for 30 sec - exiting 05:24:52 (5664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:46 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:47 (4904): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 05:53:36 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:05 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:08 (6060): No heartbeat from core client for 30 sec - exiting 06:24:09 (6060): No heartbeat from core client for 30 sec - exiting 06:24:10 (6060): No heartbeat from core client for 30 sec - exiting 06:24:11 (6060): No heartbeat from core client for 30 sec - exiting 06:24:12 (6060): No heartbeat from core client for 30 sec - exiting 06:24:13 (6060): No heartbeat from core client for 30 sec - exiting 06:24:15 (6060): No heartbeat from core client for 30 sec - exiting 06:24:16 (6060): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 06:32:53 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:55 (5860): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:05:40 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:41 (5484): No heartbeat from core client for 30 sec - exiting 07:05:42 (5484): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:28:42 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:30:38 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:31:47 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:32:53 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:38:55 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:59 (4156): No heartbeat from core client for 30 sec - exiting 07:39:00 (4156): No heartbeat from core client for 30 sec - exiting 07:39:01 (4156): No heartbeat from core client for 30 sec - exiting 07:39:02 (4156): No heartbeat from core client for 30 sec - exiting 07:39:03 (4156): No heartbeat from core client for 30 sec - exiting 07:39:04 (4156): No heartbeat from core client for 30 sec - exiting 07:39:06 (4156): No heartbeat from core client for 30 sec - exiting 07:39:07 (4156): No heartbeat from core client for 30 sec - exiting 07:39:08 (4156): No heartbeat from core client for 30 sec - exiting 07:39:09 (4156): No heartbeat from core client for 30 sec - exiting 07:39:10 (4156): No heartbeat from core client for 30 sec - exiting 07:39:11 (4156): No heartbeat from core client for 30 sec - exiting 07:39:12 (4156): No heartbeat from core client for 30 sec - exiting 07:39:13 (4156): No heartbeat from core client for 30 sec - exiting 07:39:14 (4156): No heartbeat from core client for 30 sec - exiting 07:39:15 (4156): No heartbeat from core client for 30 sec - exiting 07:39:16 (4156): No heartbeat from core client for 30 sec - exiting 07:39:18 (4156): No heartbeat from core client for 30 sec - exiting 07:39:19 (4156): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Apr 2012 10:30:44	1107891	14367426	hadcm3n_yezw_1980_40_007858717_0	25,920	46,194	1.7822