Task 16044784

Name	hadcm3n_oemj_1900_40_008474078_0
Workunit	8624917
Created	27 Sep 2013, 10:26:21 UTC
Sent	28 Sep 2013, 10:20:48 UTC
Report deadline	28 Dec 2013, 17:47:59 UTC
Received	12 Oct 2013, 15:27:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1209437
Run time	6 days 6 hours 23 min 39 sec
CPU time	5 days 22 hours 40 min 45 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	3.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 09:41:12 (7640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:41:13 (7640): No heartbeat from core client for 30 sec - exiting 09:41:14 (7640): No heartbeat from core client for 30 sec - exiting 09:41:15 (7640): No heartbeat from core client for 30 sec - exiting 09:41:16 (7640): No heartbeat from core client for 30 sec - exiting 09:41:17 (7640): No heartbeat from core client for 30 sec - exiting 09:41:19 (7640): No heartbeat from core client for 30 sec - exiting 09:41:20 (7640): No heartbeat from core client for 30 sec - exiting 09:41:21 (7640): No heartbeat from core client for 30 sec - exiting 09:41:22 (7640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 02:38:22 (11016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:23 (11016): No heartbeat from core client for 30 sec - exiting 02:38:24 (11016): No heartbeat from core client for 30 sec - exiting 02:38:25 (11016): No heartbeat from core client for 30 sec - exiting 02:38:26 (11016): No heartbeat from core client for 30 sec - exiting 02:38:27 (11016): No heartbeat from core client for 30 sec - exiting 02:38:28 (11016): No heartbeat from core client for 30 sec - exiting 02:38:30 (11016): No heartbeat from core client for 30 sec - exiting 02:38:31 (11016): No heartbeat from core client for 30 sec - exiting 02:38:32 (11016): No heartbeat from core client for 30 sec - exiting 02:38:33 (11016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:25:11 (14296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:25:12 (14296): No heartbeat from core client for 30 sec - exiting 07:25:13 (14296): No heartbeat from core client for 30 sec - exiting 07:25:15 (14296): No heartbeat from core client for 30 sec - exiting 07:25:16 (14296): No heartbeat from core client for 30 sec - exiting 07:25:17 (14296): No heartbeat from core client for 30 sec - exiting 07:25:18 (14296): No heartbeat from core client for 30 sec - exiting 07:25:19 (14296): No heartbeat from core client for 30 sec - exiting 07:25:20 (14296): No heartbeat from core client for 30 sec - exiting 07:25:21 (14296): No heartbeat from core client for 30 sec - exiting 07:25:22 (14296): No heartbeat from core client for 30 sec - exiting 06:53:19 (11388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:20 (11388): No heartbeat from core client for 30 sec - exiting 06:53:21 (11388): No heartbeat from core client for 30 sec - exiting 06:53:22 (11388): No heartbeat from core client for 30 sec - exiting 06:53:23 (11388): No heartbeat from core client for 30 sec - exiting 06:53:24 (11388): No heartbeat from core client for 30 sec - exiting 06:53:25 (11388): No heartbeat from core client for 30 sec - exiting 07:06:11 (21348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:12 (21348): No heartbeat from core client for 30 sec - exiting 07:06:13 (21348): No heartbeat from core client for 30 sec - exiting 07:06:14 (21348): No heartbeat from core client for 30 sec - exiting 07:06:15 (21348): No heartbeat from core client for 30 sec - exiting 07:06:16 (21348): No heartbeat from core client for 30 sec - exiting 07:20:09 (11352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:10 (11352): No heartbeat from core client for 30 sec - exiting 07:20:12 (11352): No heartbeat from core client for 30 sec - exiting 07:20:13 (11352): No heartbeat from core client for 30 sec - exiting 07:20:14 (11352): No heartbeat from core client for 30 sec - exiting 07:20:15 (11352): No heartbeat from core client for 30 sec - exiting 07:20:16 (11352): No heartbeat from core client for 30 sec - exiting 07:20:17 (11352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:04:47 (28712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:04:48 (28712): No heartbeat from core client for 30 sec - exiting 08:04:49 (28712): No heartbeat from core client for 30 sec - exiting 08:04:50 (28712): No heartbeat from core client for 30 sec - exiting 08:04:51 (28712): No heartbeat from core client for 30 sec - exiting 08:04:52 (28712): No heartbeat from core client for 30 sec - exiting 08:04:53 (28712): No heartbeat from core client for 30 sec - exiting 08:04:54 (28712): No heartbeat from core client for 30 sec - exiting 08:04:55 (28712): No heartbeat from core client for 30 sec - exiting 08:04:56 (28712): No heartbeat from core client for 30 sec - exiting 08:04:58 (28712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:27:40 (48828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:27:41 (48828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:17:12 (58512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:13 (58512): No heartbeat from core client for 30 sec - exiting 07:17:15 (58512): No heartbeat from core client for 30 sec - exiting 07:17:16 (58512): No heartbeat from core client for 30 sec - exiting 07:17:17 (58512): No heartbeat from core client for 30 sec - exiting 07:17:18 (58512): No heartbeat from core client for 30 sec - exiting 07:17:19 (58512): No heartbeat from core client for 30 sec - exiting 07:17:20 (58512): No heartbeat from core client for 30 sec - exiting 07:17:21 (58512): No heartbeat from core client for 30 sec - exiting 07:17:22 (58512): No heartbeat from core client for 30 sec - exiting 07:17:23 (58512): No heartbeat from core client for 30 sec - exiting 07:39:15 (64972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:16 (64972): No heartbeat from core client for 30 sec - exiting 07:39:17 (64972): No heartbeat from core client for 30 sec - exiting 07:39:18 (64972): No heartbeat from core client for 30 sec - exiting 07:39:19 (64972): No heartbeat from core client for 30 sec - exiting 07:39:20 (64972): No heartbeat from core client for 30 sec - exiting 07:39:21 (64972): No heartbeat from core client for 30 sec - exiting 07:39:22 (64972): No heartbeat from core client for 30 sec - exiting 07:39:23 (64972): No heartbeat from core client for 30 sec - exiting 07:39:24 (64972): No heartbeat from core client for 30 sec - exiting 07:50:18 (67476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:50:19 (67476): No heartbeat from core client for 30 sec - exiting 07:50:20 (67476): No heartbeat from core client for 30 sec - exiting 07:50:21 (67476): No heartbeat from core client for 30 sec - exiting 07:50:22 (67476): No heartbeat from core client for 30 sec - exiting 07:50:23 (67476): No heartbeat from core client for 30 sec - exiting 07:50:24 (67476): No heartbeat from core client for 30 sec - exiting 07:50:25 (67476): No heartbeat from core client for 30 sec - exiting 07:50:26 (67476): No heartbeat from core client for 30 sec - exiting 07:50:27 (67476): No heartbeat from core client for 30 sec - exiting 07:50:28 (67476): No heartbeat from core client for 30 sec - exiting 08:05:59 (59704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:06:00 (59704): No heartbeat from core client for 30 sec - exiting 08:06:01 (59704): No heartbeat from core client for 30 sec - exiting 08:06:02 (59704): No heartbeat from core client for 30 sec - exiting 08:06:03 (59704): No heartbeat from core client for 30 sec - exiting 08:06:04 (59704): No heartbeat from core client for 30 sec - exiting 08:20:03 (37632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:04 (37632): No heartbeat from core client for 30 sec - exiting 08:40:53 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3716, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:13:47 (9052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3340, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3340, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:22:51 (444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:35:08 (46956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:09 (47100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:42 (47968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:29 (52088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57080, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57080, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57080, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=57080, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Oct 2013 11:15:42	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	440,640	502,504	1.1404
06 Oct 2013 02:36:14	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	414,720	473,058	1.1407
05 Oct 2013 12:35:27	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	388,800	444,071	1.1422
05 Oct 2013 04:27:14	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	362,880	414,983	1.1436
04 Oct 2013 13:41:19	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	336,960	386,606	1.1473
04 Oct 2013 05:39:43	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	311,040	357,882	1.1506
03 Oct 2013 15:44:53	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	285,120	329,002	1.1539
03 Oct 2013 07:55:43	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	259,200	300,293	1.1585
02 Oct 2013 23:31:43	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	233,280	271,726	1.1648
02 Oct 2013 10:00:56	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	207,360	242,983	1.1718
02 Oct 2013 01:50:38	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	181,440	214,305	1.1811
01 Oct 2013 10:06:30	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	155,520	185,654	1.1938
01 Oct 2013 01:22:32	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	129,600	154,755	1.1941
30 Sep 2013 10:46:07	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	103,680	123,990	1.1959
30 Sep 2013 02:02:13	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	77,760	93,135	1.1977
29 Sep 2013 11:24:46	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	51,840	62,278	1.2014
29 Sep 2013 02:36:45	1209437	16044784	hadcm3n_oemj_1900_40_008474078_0	25,920	31,440	1.2130