Task 16142425

Name	hadcm3n_o7q0_1980_40_008388780_4
Workunit	8539639
Created	12 Dec 2013, 8:43:49 UTC
Sent	12 Dec 2013, 8:44:12 UTC
Report deadline	13 Mar 2014, 16:11:23 UTC
Received	19 Dec 2013, 10:20:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1194844
Run time	5 days 3 hours 20 min 42 sec
CPU time	4 days 20 hours 7 min 41 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	3.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 09:00:32 (18952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:09:16 (28180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:15:32 (38676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:56 (49800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:06 (59848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:46 (67916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:37:56 (70020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:38 (122984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:34 (133480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:25:31 (135988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:29 (151164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:10 (164168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:09:02 (197940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:54 (245388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:52 (258328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:38 (260304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:44 (260304): No heartbeat from core client for 30 sec - exiting 02:08:22 (260672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:10:31 (260164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:32 (260164): No heartbeat from core client for 30 sec - exiting 02:10:33 (260164): No heartbeat from core client for 30 sec - exiting 02:10:34 (260164): No heartbeat from core client for 30 sec - exiting 02:10:35 (260164): No heartbeat from core client for 30 sec - exiting 02:10:36 (260164): No heartbeat from core client for 30 sec - exiting 02:10:37 (260164): No heartbeat from core client for 30 sec - exiting 02:10:38 (260164): No heartbeat from core client for 30 sec - exiting 02:10:39 (260164): No heartbeat from core client for 30 sec - exiting 02:10:40 (260164): No heartbeat from core client for 30 sec - exiting 02:10:41 (260164): No heartbeat from core client for 30 sec - exiting 02:13:30 (253768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:15:09 (261728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:05 (261800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:17 (261800): No heartbeat from core client for 30 sec - exiting 02:17:18 (261800): No heartbeat from core client for 30 sec - exiting 02:17:19 (261800): No heartbeat from core client for 30 sec - exiting 02:17:59 (261432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:32 (261576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:37 (261840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:20:37 (261840): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 02:21:58 (262956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:00 (263148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:28 (263148): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 02:24:30 (262844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:43 (263476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:26:28 (263400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:26:40 (263400): No heartbeat from core client for 30 sec - exiting 02:26:41 (263400): No heartbeat from core client for 30 sec - exiting 02:27:17 (263084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:52 (263456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:04 (263456): No heartbeat from core client for 30 sec - exiting 02:28:05 (263456): No heartbeat from core client for 30 sec - exiting 02:28:06 (263456): No heartbeat from core client for 30 sec - exiting 02:28:41 (262368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:30:12 (263264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:30:53 (264316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:37 (265052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:33:05 (264348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:52 (264264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:35:05 (266204): to heartbeat from core client for 30 sec - exiting mos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:36:47 (265224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:38:43 (265764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:41 (265408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:42 (265408): No heartbeat from core client for 30 sec - exiting 02:41:23 (265360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:53 (265360): No heartbeat from core client for 30 sec - exiting 02:41:54 (265360): No heartbeat from core client for 30 sec - exiting 02:41:55 (265360): No heartbeat from core client for 30 sec - exiting 02:42:50 (266964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:44:51 (266872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:44:52 (266872): No heartbeat from core client for 30 sec - exiting 02:44:53 (266872): No heartbeat from core client for 30 sec - exiting 02:44:54 (266872): No heartbeat from core client for 30 sec - exiting 02:44:55 (266872): No heartbeat from core client for 30 sec - exiting 02:44:56 (266872): No heartbeat from core client for 30 sec - exiting 02:44:57 (266872): No heartbeat from core client for 30 sec - exiting 02:44:58 (266872): No heartbeat from core client for 30 sec - exiting 02:44:59 (266872): No heartbeat from core client for 30 sec - exiting 02:45:00 (266872): No heartbeat from core client for 30 sec - exiting 02:45:01 (266872): No heartbeat from core client for 30 sec - exiting 02:46:16 (266324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:48:21 (266852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:51:37 (267612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:38 (267612): No heartbeat from core client for 30 sec - exiting 02:51:39 (267612): No heartbeat from core client for 30 sec - exiting 02:51:40 (267612): No heartbeat from core client for 30 sec - exiting 02:51:41 (267612): No heartbeat from core client for 30 sec - exiting 02:51:42 (267612): No heartbeat from core client for 30 sec - exiting 02:51:43 (267612): No heartbeat from core client for 30 sec - exiting 02:51:44 (267612): No heartbeat from core client for 30 sec - exiting 02:51:45 (267612): No heartbeat from core client for 30 sec - exiting 02:51:46 (267612): No heartbeat from core client for 30 sec - exiting 02:51:47 (267612): No heartbeat from core client for 30 sec - exiting 02:52:42 (267972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:26 (267920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:59 (267848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:54:02 (267848): No heartbeat from core client for 30 sec - exiting 02:54:03 (267848): No heartbeat from core client for 30 sec - exiting 02:54:38 (268512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:55:03 (268512): No heartbeat from core client for 30 sec - exiting 02:55:04 (268512): No heartbeat from core client for 30 sec - exiting 02:55:05 (268512): No heartbeat from core client for 30 sec - exiting 02:55:39 (268084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:22 (269308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... A02:57:14 (268988): No heartbeat from core client for 30 sec - exiting tmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - No 'heartbeat' from BOINC... 02:57:27 (268988): No heartbeat from core client for 30 sec - exiting 02:57:28 (268988): No heartbeat from core client for 30 sec - exiting 02:57:29 (268988): No heartbeat from core client for 30 sec - exiting 02:57:30 (268988): No heartbeat from core client for 30 sec - exiting 02:57:31 (268988): No heartbeat from core client for 30 sec - exiting 02:57:32 (268988): No heartbeat from core client for 30 sec - exiting 02:57:33 (268988): No heartbeat from core client for 30 sec - exiting 02:58:49 (262360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:22 (269100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:00:28 (268752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:02 (268364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:45 (270104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:04:43 (269920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:29 (268428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:11 (270388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:29 (270332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:44 (270504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:28 (271920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:09 (271888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:09 (267360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:26:01 (266548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:37 (279932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:17 (283592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:38 (278720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:00 (278568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=285700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=285700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=285700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=285700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=285700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=285700, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2013 00:32:49	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	414,720	416,112	1.0034
18 Dec 2013 16:53:47	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	388,800	389,855	1.0027
18 Dec 2013 09:22:06	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	362,880	363,679	1.0022
18 Dec 2013 01:59:38	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	336,960	337,851	1.0026
17 Dec 2013 18:21:41	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	311,040	311,514	1.0015
17 Dec 2013 10:47:06	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	285,120	285,085	0.9999
17 Dec 2013 03:09:22	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	259,200	258,761	0.9983
16 Dec 2013 19:32:00	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	233,280	232,781	0.9979
16 Dec 2013 12:08:29	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	207,360	206,844	0.9975
16 Dec 2013 04:37:26	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	181,440	181,057	0.9979
15 Dec 2013 10:06:46	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	155,520	154,886	0.9959
14 Dec 2013 11:46:40	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	129,600	129,039	0.9957
14 Dec 2013 04:24:18	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	103,680	103,398	0.9973
13 Dec 2013 20:56:48	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	77,760	77,612	0.9981
13 Dec 2013 11:53:58	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	51,840	51,869	1.0006
13 Dec 2013 04:52:16	1194844	16142425	hadcm3n_o7q0_1980_40_008388780_4	25,920	25,964	1.0017