Task 16297995

Name	hadcm3n_7e25_1980_40_008429504_3
Workunit	8580360
Created	3 Mar 2014, 10:56:33 UTC
Sent	3 Mar 2014, 10:56:55 UTC
Report deadline	4 Sep 2023, 16:16:55 UTC
Received	12 Mar 2014, 14:15:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1308965
Run time	2 days 10 hours 46 min 31 sec
CPU time	2 days 1 hours 58 min 56 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	3.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 19:24:27 (208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:30:25 (10820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:16:12 (18956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:02:31 (20356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:04:28 (18276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:30:31 (17796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:31 (14984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:07 (20996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:52 (19120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:52:29 (20216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:52 (20808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:10 (21236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 20:56:24 (12488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:24 (7024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:58:00 (21064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:59:21 (11668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:00:23 (10472): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 21:35:03 (19944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:41:47 (9512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:46:59 (16448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:48:19 (18828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:01 (17848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:52 (16968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 22:27:48 (21408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:41 (13596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19836, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19836, iMonCtr=1 Model crash detected, will try to restart... 08:30:29 (19836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:39:26 (25612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22476, iMonCtr=1 Model crash detected, will try to restart... 10:21:32 (22476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:48:40 (22532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:52:20 (29692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:31 (31376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:10:23 (29848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:27:12 (29056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40936, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40936, iMonCtr=1 Model crash detected, will try to restart... 09:41:27 (40936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:19 (12508): No heartbeat from core client for 30 sec - exiting 09:42:20 (12508): No heartbeat from core client for 30 sec - exiting 09:42:21 (12508): No heartbeat from core client for 30 sec - exiting 09:42:22 (12508): No heartbeat from core client for 30 sec - exiting 09:42:23 (12508): No heartbeat from core client for 30 sec - exiting 09:42:24 (12508): No heartbeat from core client for 30 sec - exiting 09:42:25 (12508): No heartbeat from core client for 30 sec - exiting 09:42:26 (12508): No heartbeat from core client for 30 sec - exiting 09:42:27 (12508): No heartbeat from core client for 30 sec - exiting 09:42:28 (12508): No heartbeat from core client for 30 sec - exiting 09:42:29 (12508): No heartbeat from core client for 30 sec - exiting 09:42:30 (12508): No heartbeat from core client for 30 sec - exiting 09:42:31 (12508): No heartbeat from core client for 30 sec - exiting 09:42:32 (12508): No heartbeat from core client for 30 sec - exiting 09:42:33 (12508): No heartbeat from core client for 30 sec - exiting 09:42:34 (12508): No heartbeat from core client for 30 sec - exiting 09:42:35 (12508): No heartbeat from core client for 30 sec - exiting 09:42:36 (12508): No heartbeat from core client for 30 sec - exiting 09:42:37 (12508): No heartbeat from core client for 30 sec - exiting 09:42:38 (12508): No heartbeat from core client for 30 sec - exiting 09:42:39 (12508): No heartbeat from core client for 30 sec - exiting 09:42:40 (12508): No heartbeat from core client for 30 sec - exiting 09:42:41 (12508): No heartbeat from core client for 30 sec - exiting 09:42:42 (12508): No heartbeat from core client for 30 sec - exiting 09:42:43 (12508): No heartbeat from core client for 30 sec - exiting 09:42:44 (12508): No heartbeat from core client for 30 sec - exiting 09:42:45 (12508): No heartbeat from core client for 30 sec - exiting 09:42:46 (12508): No heartbeat from core client for 30 sec - exiting 09:42:47 (12508): No heartbeat from core client for 30 sec - exiting 09:42:48 (12508): No heartbeat from core client for 30 sec - exiting 09:42:49 (12508): No heartbeat from core client for 30 sec - exiting 09:42:50 (12508): No heartbeat from core client for 30 sec - exiting 09:42:51 (12508): No heartbeat from core client for 30 sec - exiting 09:42:52 (12508): No heartbeat from core client for 30 sec - exiting 09:42:53 (12508): No heartbeat from core client for 30 sec - exiting 09:42:54 (12508): No heartbeat from core client for 30 sec - exiting 09:42:55 (12508): No heartbeat from core client for 30 sec - exiting 09:42:56 (12508): No heartbeat from core client for 30 sec - exiting 09:42:57 (12508): No heartbeat from core client for 30 sec - exiting 09:42:58 (12508): No heartbeat from core client for 30 sec - exiting 09:42:59 (12508): No heartbeat from core client for 30 sec - exiting 09:43:00 (12508): No heartbeat from core client for 30 sec - exiting 09:43:01 (12508): No heartbeat from core client for 30 sec - exiting 09:43:02 (12508): No heartbeat from core client for 30 sec - exiting 09:43:03 (12508): No heartbeat from core client for 30 sec - exiting 09:43:04 (12508): No heartbeat from core client for 30 sec - exiting 09:43:05 (12508): No heartbeat from core client for 30 sec - exiting 09:43:06 (12508): No heartbeat from core client for 30 sec - exiting 09:43:07 (12508): No heartbeat from core client for 30 sec - exiting 09:43:08 (12508): No heartbeat from core client for 30 sec - exiting 09:43:09 (12508): No heartbeat from core client for 30 sec - exiting 09:43:10 (12508): No heartbeat from core client for 30 sec - exiting 09:43:11 (12508): No heartbeat from core client for 30 sec - exiting 09:43:12 (12508): No heartbeat from core client for 30 sec - exiting 09:43:13 (12508): No heartbeat from core client for 30 sec - exiting 09:43:14 (12508): No heartbeat from core client for 30 sec - exiting 09:43:15 (12508): No heartbeat from core client for 30 sec - exiting 09:43:16 (12508): No heartbeat from core client for 30 sec - exiting 09:43:17 (12508): No heartbeat from core client for 30 sec - exiting 09:43:18 (12508): No heartbeat from core client for 30 sec - exiting 09:43:19 (12508): No heartbeat from core client for 30 sec - exiting 09:43:20 (12508): No heartbeat from core client for 30 sec - exiting 09:43:21 (12508): No heartbeat from core client for 30 sec - exiting 09:43:22 (12508): No heartbeat from core client for 30 sec - exiting 09:43:23 (12508): No heartbeat from core client for 30 sec - exiting 09:43:24 (12508): No heartbeat from core client for 30 sec - exiting 09:43:25 (12508): No heartbeat from core client for 30 sec - exiting 09:43:26 (12508): No heartbeat from core client for 30 sec - exiting 09:43:27 (12508): No heartbeat from core client for 30 sec - exiting 09:43:28 (12508): No heartbeat from core client for 30 sec - exiting 09:43:29 (12508): No heartbeat from core client for 30 sec - exiting 09:43:34 (12508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:41:39 (36076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=33444, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Mar 2014 19:08:07	1308965	16297995	hadcm3n_7e25_1980_40_008429504_3	155,520	172,823	1.1113
05 Mar 2014 09:38:27	1308965	16297995	hadcm3n_7e25_1980_40_008429504_3	129,600	144,274	1.1132
05 Mar 2014 00:56:06	1308965	16297995	hadcm3n_7e25_1980_40_008429504_3	103,680	116,156	1.1203
04 Mar 2014 15:49:07	1308965	16297995	hadcm3n_7e25_1980_40_008429504_3	77,760	88,300	1.1355
04 Mar 2014 06:33:56	1308965	16297995	hadcm3n_7e25_1980_40_008429504_3	51,840	60,159	1.1605
03 Mar 2014 20:51:56	1308965	16297995	hadcm3n_7e25_1980_40_008429504_3	25,920	32,174	1.2413