Task 15427218

Name	hadcm3n_zfm0_1920_40_008244169_0
Workunit	8399293
Created	5 Nov 2012, 14:18:37 UTC
Sent	5 Nov 2012, 14:18:42 UTC
Report deadline	4 Feb 2013, 21:45:53 UTC
Received	7 Nov 2012, 20:25:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1247100
Run time	15 hours 13 min 57 sec
CPU time	6 hours 50 min 5 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 09:02:11 (6836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:02:26 (6836): No heartbeat from core client for 30 sec - exiting 09:02:29 (6836): No heartbeat from core client for 30 sec - exiting 09:02:30 (6836): No heartbeat from core client for 30 sec - exiting 09:02:31 (6836): No heartbeat from core client for 30 sec - exiting 09:02:32 (6836): No heartbeat from core client for 30 sec - exiting 09:02:35 (6836): No heartbeat from core client for 30 sec - exiting 09:02:36 (6836): No heartbeat from core client for 30 sec - exiting 09:02:37 (6836): No heartbeat from core client for 30 sec - exiting 09:02:39 (6836): No heartbeat from core client for 30 sec - exiting 09:02:40 (6836): No heartbeat from core client for 30 sec - exiting 09:02:42 (6836): No heartbeat from core client for 30 sec - exiting 09:02:43 (6836): No heartbeat from core client for 30 sec - exiting 09:02:44 (6836): No heartbeat from core client for 30 sec - exiting 09:02:45 (6836): No heartbeat from core client for 30 sec - exiting 09:02:46 (6836): No heartbeat from core client for 30 sec - exiting 09:02:47 (6836): No heartbeat from core client for 30 sec - exiting 09:02:48 (6836): No heartbeat from core client for 30 sec - exiting 09:15:42 (132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:58 (132): No heartbeat from core client for 30 sec - exiting 09:22:13 (132): No heartbeat from core client for 30 sec - exiting 09:22:17 (132): No heartbeat from core client for 30 sec - exiting 09:22:18 (132): No heartbeat from core client for 30 sec - exiting 09:22:19 (132): No heartbeat from core client for 30 sec - exiting 09:22:21 (132): No heartbeat from core client for 30 sec - exiting 09:22:23 (132): No heartbeat from core client for 30 sec - exiting 09:22:25 (132): No heartbeat from core client for 30 sec - exiting 09:22:27 (132): No heartbeat from core client for 30 sec - exiting 09:22:29 (132): No heartbeat from core client for 30 sec - exiting 09:22:31 (132): No heartbeat from core client for 30 sec - exiting 09:22:33 (132): No heartbeat from core client for 30 sec - exiting 09:22:35 (132): No heartbeat from core client for 30 sec - exiting 09:22:37 (132): No heartbeat from core client for 30 sec - exiting 11:10:39 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:10:49 (5800): No heartbeat from core client for 30 sec - exiting 11:10:50 (5800): No heartbeat from core client for 30 sec - exiting 11:10:52 (5800): No heartbeat from core client for 30 sec - exiting 11:10:54 (5800): No heartbeat from core client for 30 sec - exiting 11:10:56 (5800): No heartbeat from core client for 30 sec - exiting 11:10:57 (5800): No heartbeat from core client for 30 sec - exiting 11:10:58 (5800): No heartbeat from core client for 30 sec - exiting 11:24:39 (7064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:23 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:35 (6476): No heartbeat from core client for 30 sec - exiting 11:50:35 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:48 (4568): No heartbeat from core client for 30 sec - exiting 11:50:51 (4568): No heartbeat from core client for 30 sec - exiting 11:50:53 (4568): No heartbeat from core client for 30 sec - exiting 11:50:55 (4568): No heartbeat from core client for 30 sec - exiting 11:50:57 (4568): No heartbeat from core client for 30 sec - exiting 11:59:50 (3388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:47 (6856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:26 (196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:31 (196): No heartbeat from core client for 30 sec - exiting 12:36:34 (196): No heartbeat from core client for 30 sec - exiting 12:36:35 (196): No heartbeat from core client for 30 sec - exiting 12:36:37 (196): No heartbeat from core client for 30 sec - exiting 12:36:38 (196): No heartbeat from core client for 30 sec - exiting 12:36:39 (196): No heartbeat from core client for 30 sec - exiting 12:36:40 (196): No heartbeat from core client for 30 sec - exiting 12:45:49 (7040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:46:54 (7040): No heartbeat from core client for 30 sec - exiting 12:46:55 (7040): No heartbeat from core client for 30 sec - exiting 12:46:57 (7040): No heartbeat from core client for 30 sec - exiting 12:46:59 (7040): No heartbeat from core client for 30 sec - exiting 12:47:01 (7040): No heartbeat from core client for 30 sec - exiting 12:47:03 (7040): No heartbeat from core client for 30 sec - exiting 12:47:05 (7040): No heartbeat from core client for 30 sec - exiting 12:47:07 (7040): No heartbeat from core client for 30 sec - exiting 12:47:09 (7040): No heartbeat from core client for 30 sec - exiting 12:57:15 (4268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:17:59 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:09 (5372): No heartbeat from core client for 30 sec - exiting 13:18:10 (5372): No heartbeat from core client for 30 sec - exiting 13:18:11 (5372): No heartbeat from core client for 30 sec - exiting 13:18:13 (5372): No heartbeat from core client for 30 sec - exiting 14:57:19 (572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:26 (572): No heartbeat from core client for 30 sec - exiting 14:57:28 (572): No heartbeat from core client for 30 sec - exiting 14:57:30 (572): No heartbeat from core client for 30 sec - exiting 14:57:32 (572): No heartbeat from core client for 30 sec - exiting 14:57:34 (572): No heartbeat from core client for 30 sec - exiting 14:57:36 (572): No heartbeat from core client for 30 sec - exiting 14:57:38 (572): No heartbeat from core client for 30 sec - exiting 14:57:39 (572): No heartbeat from core client for 30 sec - exiting 14:57:40 (572): No heartbeat from core client for 30 sec - exiting 14:57:41 (572): No heartbeat from core client for 30 sec - exiting 14:57:42 (572): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:38:13 (1128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:20 (1128): No heartbeat from core client for 30 sec - exiting 15:38:23 (1128): No heartbeat from core client for 30 sec - exiting 15:38:24 (1128): No heartbeat from core client for 30 sec - exiting 15:38:26 (1128): No heartbeat from core client for 30 sec - exiting 15:38:27 (1128): No heartbeat from core client for 30 sec - exiting 15:38:28 (1128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:20:33 (6668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:01 (6500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:09 (6500): No heartbeat from core client for 30 sec - exiting 04:36:10 (6500): No heartbeat from core client for 30 sec - exiting 04:36:12 (6500): No heartbeat from core client for 30 sec - exiting 04:36:14 (6500): No heartbeat from core client for 30 sec - exiting 04:36:15 (6500): No heartbeat from core client for 30 sec - exiting 04:43:29 (5980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:29:48 (7544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:30:01 (7544): No heartbeat from core client for 30 sec - exiting 05:30:02 (7544): No heartbeat from core client for 30 sec - exiting 05:30:03 (7544): No heartbeat from core client for 30 sec - exiting 05:30:05 (7544): No heartbeat from core client for 30 sec - exiting 05:30:06 (7544): No heartbeat from core client for 30 sec - exiting 05:30:07 (7544): No heartbeat from core client for 30 sec - exiting 05:30:09 (7544): No heartbeat from core client for 30 sec - exiting 05:30:11 (7544): No heartbeat from core client for 30 sec - exiting 05:30:12 (7544): No heartbeat from core client for 30 sec - exiting 05:30:13 (7544): No heartbeat from core client for 30 sec - exiting 05:40:39 (6736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:41 (6736): No heartbeat from core client for 30 sec - exiting 05:40:42 (6736): No heartbeat from core client for 30 sec - exiting 05:40:44 (6736): No heartbeat from core client for 30 sec - exiting 05:45:47 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:10 (6664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:13 (6664): No heartbeat from core client for 30 sec - exiting 06:00:15 (6664): No heartbeat from core client for 30 sec - exiting 06:00:16 (6664): No heartbeat from core client for 30 sec - exiting 06:00:18 (6664): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 06:06:20 (6864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:06:26 (6864): No heartbeat from core client for 30 sec - exiting 06:14:31 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:21 (8116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:34 (8116): No heartbeat from core client for 30 sec - exiting 07:03:35 (8116): No heartbeat from core client for 30 sec - exiting 07:44:12 (7484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:25 (7484): No heartbeat from core client for 30 sec - exiting 07:44:28 (7484): No heartbeat from core client for 30 sec - exiting 07:44:29 (7484): No heartbeat from core client for 30 sec - exiting 07:44:30 (7484): No heartbeat from core client for 30 sec - exiting 07:44:31 (7484): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Nov 2012 14:33:23	1247100	15427218	hadcm3n_zfm0_1920_40_008244169_0	25,920	46,297	1.7861