Task 15724851

Name	hadcm3n_4anl_1980_40_008324092_1
Workunit	8475227
Created	15 Apr 2013, 5:13:33 UTC
Sent	15 Apr 2013, 5:13:37 UTC
Report deadline	15 Jul 2013, 12:40:48 UTC
Received	28 Apr 2013, 19:04:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1080358
Run time	10 days 5 hours 32 min 58 sec
CPU time	9 days 10 hours 52 min 37 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:21:33 (5588): No heartbeat from core client for 30 sec - exiting 17:21:34 (5588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... 20:17:53 (4840): No heartbeat from core client for 30 sec - exiting 20:17:54 (4840): No heartbeat from core client for 30 sec - exiting 20:17:55 (4840): No heartbeat from core client for 30 sec - exiting 20:17:56 (4840): No heartbeat from core client for 30 sec - exiting 20:17:57 (4840): No heartbeat from core client for 30 sec - exiting 20:17:59 (4840): No heartbeat from core client for 30 sec - exiting 20:18:00 (4840): No heartbeat from core client for 30 sec - exiting 20:18:01 (4840): No heartbeat from core client for 30 sec - exiting 20:18:02 (4840): No heartbeat from core client for 30 sec - exiting 20:18:03 (4840): No heartbeat from core client for 30 sec - exiting 20:18:04 (4840): No heartbeat from core client for 30 sec - exiting 20:18:05 (4840): No heartbeat from core client for 30 sec - exiting 20:18:06 (4840): No heartbeat from core client for 30 sec - exiting 20:18:07 (4840): No heartbeat from core client for 30 sec - exiting 20:18:08 (4840): No heartbeat from core client for 30 sec - exiting 20:18:09 (4840): No heartbeat from core client for 30 sec - exiting 20:18:11 (4840): No heartbeat from core client for 30 sec - exiting 20:18:12 (4840): No heartbeat from core client for 30 sec - exiting 20:18:45 (4840): No heartbeat from core client for 30 sec - exiting 20:18:47 (4840): No heartbeat from core client for 30 sec - exiting 20:18:48 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:24 (2192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:26 (1724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:21:59 (5680): No heartbeat from core client for 30 sec - exiting 06:22:33 (5680): No heartbeat from core client for 30 sec - exiting 06:22:34 (5680): No heartbeat from core client for 30 sec - exiting 06:22:35 (5680): No heartbeat from core client for 30 sec - exiting 06:22:36 (5680): No heartbeat from core client for 30 sec - exiting 06:22:37 (5680): No heartbeat from core client for 30 sec - exiting 06:22:38 (5680): No heartbeat from core client for 30 sec - exiting 06:22:39 (5680): No heartbeat from core client for 30 sec - exiting 06:22:41 (5680): No heartbeat from core client for 30 sec - exiting 06:22:42 (5680): No heartbeat from core client for 30 sec - exiting 06:22:43 (5680): No heartbeat from core client for 30 sec - exiting 06:22:44 (5680): No heartbeat from core client for 30 sec - exiting 06:22:45 (5680): No heartbeat from core client for 30 sec - exiting 06:22:46 (5680): No heartbeat from core client for 30 sec - exiting 06:22:47 (5680): No heartbeat from core client for 30 sec - exiting 06:22:48 (5680): No heartbeat from core client for 30 sec - exiting 06:22:49 (5680): No heartbeat from core client for 30 sec - exiting 06:22:50 (5680): No heartbeat from core client for 30 sec - exiting 06:22:51 (5680): No heartbeat from core client for 30 sec - exiting 06:22:53 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:23:55 (2668): No heartbeat from core client for 30 sec - exiting 06:23:56 (2668): No heartbeat from core client for 30 sec - exiting 06:23:57 (2668): No heartbeat from core client for 30 sec - exiting 06:23:58 (2668): No heartbeat from core client for 30 sec - exiting 06:23:59 (2668): No heartbeat from core client for 30 sec - exiting 06:24:01 (2668): No heartbeat from core client for 30 sec - exiting 06:24:02 (2668): No heartbeat from core client for 30 sec - exiting 06:24:03 (2668): No heartbeat from core client for 30 sec - exiting 06:24:04 (2668): No heartbeat from core client for 30 sec - exiting 06:24:05 (2668): No heartbeat from core client for 30 sec - exiting 06:24:06 (2668): No heartbeat from core client for 30 sec - exiting 06:24:07 (2668): No heartbeat from core client for 30 sec - exiting 06:24:08 (2668): No heartbeat from core client for 30 sec - exiting 06:24:09 (2668): No heartbeat from core client for 30 sec - exiting 06:24:10 (2668): No heartbeat from core client for 30 sec - exiting 06:24:11 (2668): No heartbeat from core client for 30 sec - exiting 06:24:13 (2668): No heartbeat from core client for 30 sec - exiting 06:24:14 (2668): No heartbeat from core client for 30 sec - exiting 06:24:15 (2668): No heartbeat from core client for 30 sec - exiting 06:24:16 (2668): No heartbeat from core client for 30 sec - exiting 06:24:17 (2668): No heartbeat from core client for 30 sec - exiting 06:24:18 (2668): No heartbeat from core client for 30 sec - exiting 06:24:19 (2668): No heartbeat from core client for 30 sec - exiting 06:24:20 (2668): No heartbeat from core client for 30 sec - exiting 06:24:21 (2668): No heartbeat from core client for 30 sec - exiting 06:24:22 (2668): No heartbeat from core client for 30 sec - exiting 06:24:23 (2668): No heartbeat from core client for 30 sec - exiting 06:24:25 (2668): No heartbeat from core client for 30 sec - exiting 06:24:26 (2668): No heartbeat from core client for 30 sec - exiting 06:24:27 (2668): No heartbeat from core client for 30 sec - exiting 06:24:28 (2668): No heartbeat from core client for 30 sec - exiting 06:24:29 (2668): No heartbeat from core client for 30 sec - exiting 06:24:30 (2668): No heartbeat from core client for 30 sec - exiting 06:24:31 (2668): No heartbeat from core client for 30 sec - exiting 06:24:32 (2668): No heartbeat from core client for 30 sec - exiting 06:24:33 (2668): No heartbeat from core client for 30 sec - exiting 06:24:34 (2668): No heartbeat from core client for 30 sec - exiting 06:24:35 (2668): No heartbeat from core client for 30 sec - exiting 06:24:37 (2668): No heartbeat from core client for 30 sec - exiting 06:24:38 (2668): No heartbeat from core client for 30 sec - exiting 06:24:39 (2668): No heartbeat from core client for 30 sec - exiting 06:24:40 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:28:50 (5512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:18:17 (6928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:42 (2976): No heartbeat from core client for 30 sec - exiting 13:00:43 (2976): No heartbeat from core client for 30 sec - exiting 13:00:44 (2976): No heartbeat from core client for 30 sec - exiting 13:00:45 (2976): No heartbeat from core client for 30 sec - exiting 13:00:46 (2976): No heartbeat from core client for 30 sec - exiting 13:00:47 (2976): No heartbeat from core client for 30 sec - exiting 13:00:48 (2976): No heartbeat from core client for 30 sec - exiting 13:00:49 (2976): No heartbeat from core client for 30 sec - exiting 13:00:50 (2976): No heartbeat from core client for 30 sec - exiting 13:00:51 (2976): No heartbeat from core client for 30 sec - exiting 13:00:52 (2976): No heartbeat from core client for 30 sec - exiting 13:00:53 (2976): No heartbeat from core client for 30 sec - exiting 13:00:54 (2976): No heartbeat from core client for 30 sec - exiting 13:00:55 (2976): No heartbeat from core client for 30 sec - exiting 13:00:56 (2976): No heartbeat from core client for 30 sec - exiting 13:00:57 (2976): No heartbeat from core client for 30 sec - exiting 13:00:58 (2976): No heartbeat from core client for 30 sec - exiting 13:00:59 (2976): No heartbeat from core client for 30 sec - exiting 13:01:00 (2976): No heartbeat from core client for 30 sec - exiting 13:01:01 (2976): No heartbeat from core client for 30 sec - exiting 13:01:02 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:03 (2976): No heartbeat from core client for 30 sec - exiting 13:01:04 (2976): No heartbeat from core client for 30 sec - exiting 13:01:06 (2976): No heartbeat from core client for 30 sec - exiting Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 13:02:34 (3156): No heartbeat from core client for 30 sec - exiting 13:02:35 (3156): No heartbeat from core client for 30 sec - exiting 13:02:36 (3156): No heartbeat from core client for 30 sec - exiting 13:02:37 (3156): No heartbeat from core client for 30 sec - exiting 13:02:39 (3156): No heartbeat from core client for 30 sec - exiting 13:02:40 (3156): No heartbeat from core client for 30 sec - exiting 13:02:41 (3156): No heartbeat from core client for 30 sec - exiting 13:02:42 (3156): No heartbeat from core client for 30 sec - exiting 13:02:43 (3156): No heartbeat from core client for 30 sec - exiting 13:02:44 (3156): No heartbeat from core client for 30 sec - exiting 13:02:45 (3156): No heartbeat from core client for 30 sec - exiting 13:02:46 (3156): No heartbeat from core client for 30 sec - exiting 13:02:47 (3156): No heartbeat from core client for 30 sec - exiting 13:02:48 (3156): No heartbeat from core client for 30 sec - exiting 13:02:49 (3156): No heartbeat from core client for 30 sec - exiting 13:02:51 (3156): No heartbeat from core client for 30 sec - exiting 13:02:52 (3156): No heartbeat from core client for 30 sec - exiting 13:02:53 (3156): No heartbeat from core client for 30 sec - exiting 13:02:54 (3156): No heartbeat from core client for 30 sec - exiting 13:02:55 (3156): No heartbeat from core client for 30 sec - exiting 13:02:56 (3156): No heartbeat from core client for 30 sec - exiting 13:02:57 (3156): No heartbeat from core client for 30 sec - exiting 13:02:58 (3156): No heartbeat from core client for 30 sec - exiting 13:02:59 (3156): No heartbeat from core client for 30 sec - exiting 13:03:00 (3156): No heartbeat from core client for 30 sec - exiting 13:03:01 (3156): No heartbeat from core client for 30 sec - exiting 13:03:03 (3156): No heartbeat from core client for 30 sec - exiting 13:03:04 (3156): No heartbeat from core client for 30 sec - exiting 13:03:05 (3156): No heartbeat from core client for 30 sec - exiting 13:03:06 (3156): No heartbeat from core client for 30 sec - exiting 13:03:07 (3156): No heartbeat from core client for 30 sec - exiting 13:03:08 (3156): No heartbeat from core client for 30 sec - exiting 13:03:09 (3156): No heartbeat from core client for 30 sec - exiting Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 13:03:10 (3156): No heartbeat from core client for 30 sec - exiting 13:03:11 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Apr 2013 05:30:09	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	440,640	783,686	1.7785
27 Apr 2013 15:30:48	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	414,720	737,531	1.7784
27 Apr 2013 01:40:50	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	388,800	691,298	1.7780
26 Apr 2013 07:22:24	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	362,880	647,663	1.7848
25 Apr 2013 16:51:23	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	336,960	602,465	1.7879
25 Apr 2013 02:55:02	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	311,040	556,481	1.7891
24 Apr 2013 12:41:24	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	285,120	511,889	1.7953
23 Apr 2013 21:35:50	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	259,200	465,590	1.7963
23 Apr 2013 06:43:24	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	233,280	418,683	1.7948
22 Apr 2013 13:35:06	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	207,360	372,287	1.7954
21 Apr 2013 19:21:57	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	181,440	325,703	1.7951
21 Apr 2013 04:45:57	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	155,520	280,174	1.8015
20 Apr 2013 13:20:29	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	129,600	233,660	1.8029
19 Apr 2013 22:50:24	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	103,680	187,365	1.8071
18 Apr 2013 16:15:59	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	77,760	140,784	1.8105
17 Apr 2013 19:00:13	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	51,840	94,555	1.8240
16 Apr 2013 04:08:13	1080358	15724851	hadcm3n_4anl_1980_40_008324092_1	25,920	47,187	1.8205