Task 15940664

Name	hadcm3n_n65v_1880_40_008396317_1
Workunit	8547176
Created	26 Aug 2013, 8:47:51 UTC
Sent	26 Aug 2013, 8:48:46 UTC
Report deadline	25 Nov 2013, 16:15:57 UTC
Received	7 Nov 2013, 17:42:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	1 days 8 hours 21 min 35 sec
CPU time	1 days 0 hours 29 min 41 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	3.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 02:58:34 (82336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:22 (20952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:37 (50600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:37:15 (62084): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 05:37:16 (62084): No heartbeat from core client for 30 sec - exiting 05:39:34 (13788): No heartbeat from core client for 30 sec - exiting 05:39:40 (13788): No heartbeat from core client for 30 sec - exiting 05:39:41 (13788): No heartbeat from core client for 30 sec - exiting 05:39:42 (13788): No heartbeat from core client for 30 sec - exiting 05:39:43 (13788): No heartbeat from core client for 30 sec - exiting 05:39:45 (13788): No heartbeat from core client for 30 sec - exiting 05:39:46 (13788): No heartbeat from core client for 30 sec - exiting 05:39:47 (13788): No heartbeat from core client for 30 sec - exiting 05:39:48 (13788): No heartbeat from core client for 30 sec - exiting 05:39:49 (13788): No heartbeat from core client for 30 sec - exiting 05:39:50 (13788): No heartbeat from core client for 30 sec - exiting 05:39:51 (13788): No heartbeat from core client for 30 sec - exiting 05:39:53 (13788): No heartbeat from core client for 30 sec - exiting 05:39:54 (13788): No heartbeat from core client for 30 sec - exiting 05:39:55 (13788): No heartbeat from core client for 30 sec - exiting 05:39:56 (13788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:39 (26032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:40 (26032): No heartbeat from core client for 30 sec - exiting 05:41:41 (26032): No heartbeat from core client for 30 sec - exiting 05:41:43 (26032): No heartbeat from core client for 30 sec - exiting 05:41:44 (26032): No heartbeat from core client for 30 sec - exiting 05:45:07 (25016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:45:18 (25016): No heartbeat from core client for 30 sec - exiting 05:45:22 (25016): No heartbeat from core client for 30 sec - exiting 05:58:43 (79952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:40:54 (73080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:56 (73080): No heartbeat from core client for 30 sec - exiting 10:40:57 (73080): No heartbeat from core client for 30 sec - exiting 10:40:58 (73080): No heartbeat from core client for 30 sec - exiting 10:40:59 (73080): No heartbeat from core client for 30 sec - exiting 10:41:00 (73080): No heartbeat from core client for 30 sec - exiting 10:41:01 (73080): No heartbeat from core client for 30 sec - exiting 10:54:19 (73036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:12 (74396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 12:28:02 (73736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:28:03 (73736): No heartbeat from core client for 30 sec - exiting 12:28:04 (73736): No heartbeat from core client for 30 sec - exiting 12:28:05 (73736): No heartbeat from core client for 30 sec - exiting 12:35:17 (74588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:53:14 (6232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 16:57:11 (4620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:57:12 (4620): No heartbeat from core client for 30 sec - exiting 16:57:13 (4620): No heartbeat from core client for 30 sec - exiting 16:57:14 (4620): No heartbeat from core client for 30 sec - exiting 16:57:15 (4620): No heartbeat from core client for 30 sec - exiting 16:57:16 (4620): No heartbeat from core client for 30 sec - exiting 16:57:17 (4620): No heartbeat from core client for 30 sec - exiting 16:57:18 (4620): No heartbeat from core client for 30 sec - exiting 16:57:19 (4620): No heartbeat from core client for 30 sec - exiting 17:02:32 (9440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:02 (1760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:06 (1760): No heartbeat from core client for 30 sec - exiting 17:20:07 (1760): No heartbeat from core client for 30 sec - exiting 17:20:08 (1760): No heartbeat from core client for 30 sec - exiting 17:20:09 (1760): No heartbeat from core client for 30 sec - exiting 17:20:10 (1760): No heartbeat from core client for 30 sec - exiting 17:20:11 (1760): No heartbeat from core client for 30 sec - exiting 17:39:39 (1312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:30 (7048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:06 (9728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:35 (10276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:38 (10276): No heartbeat from core client for 30 sec - exiting 19:33:39 (10276): No heartbeat from core client for 30 sec - exiting 20:15:26 (10888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:29 (10888): No heartbeat from core client for 30 sec - exiting 20:20:09 (11912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:19:57 (28496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:12:52 (9248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 11:59:52 (7640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:59:53 (7640): No heartbeat from core client for 30 sec - exiting 11:59:54 (7640): No heartbeat from core client for 30 sec - exiting 11:59:55 (7640): No heartbeat from core client for 30 sec - exiting 11:59:56 (7640): No heartbeat from core client for 30 sec - exiting 12:00:00 (7640): No heartbeat from core client for 30 sec - exiting 12:00:01 (7640): No heartbeat from core client for 30 sec - exiting 13:21:32 (12576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:29:25 (32516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:51 (7252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:59 (7252): No heartbeat from core client for 30 sec - exiting 19:45:00 (7252): No heartbeat from core client for 30 sec - exiting 19:45:01 (7252): No heartbeat from core client for 30 sec - exiting 19:45:02 (7252): No heartbeat from core client for 30 sec - exiting 19:45:03 (7252): No heartbeat from core client for 30 sec - exiting 19:45:04 (7252): No heartbeat from core client for 30 sec - exiting 19:45:05 (7252): No heartbeat from core client for 30 sec - exiting 19:45:06 (7252): No heartbeat from core client for 30 sec - exiting 19:45:07 (7252): No heartbeat from core client for 30 sec - exiting 19:45:08 (7252): No heartbeat from core client for 30 sec - exiting 21:07:51 (38640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:27 (30476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:28 (30476): No heartbeat from core client for 30 sec - exiting 21:38:29 (30476): No heartbeat from core client for 30 sec - exiting 21:38:30 (30476): No heartbeat from core client for 30 sec - exiting 21:46:49 (22600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:45:50 (10044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:51:20 (8076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:51:27 (8076): No heartbeat from core client for 30 sec - exiting 02:13:28 (35596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:51:45 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:31 (32396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:26 (37124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:32 (37124): No heartbeat from core client for 30 sec - exiting 08:32:33 (37124): No heartbeat from core client for 30 sec - exiting 08:32:34 (37124): No heartbeat from core client for 30 sec - exiting 09:53:25 (22720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:21 (876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:02:58 (32540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:03 (36896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:17:05 (10928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:49 (37592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:35:06 (31500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:29:37 (8296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:29:43 (8296): No heartbeat from core client for 30 sec - exiting 16:29:44 (8296): No heartbeat from core client for 30 sec - exiting 16:35:57 (1908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:37 (29400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:23 (11456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:50:35 (10856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:04 (16816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:34 (32008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:50 (22024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:10 (41300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:09:28 (12472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:48 (40040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40740, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=40740, iMonCtr=1 Model crash detected, will try to restart... 02:22:16 (40740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=41492, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 11:15:43 (8020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:09 (1956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9116, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9116, iMonCtr=1 Model crash detected, will try to restart... 11:34:20 (9116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13288, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2013 05:19:15	1237173	15940664	hadcm3n_n65v_1880_40_008396317_1	77,760	86,459	1.1119
09 Oct 2013 17:00:54	1237173	15940664	hadcm3n_n65v_1880_40_008396317_1	51,840	57,823	1.1154
03 Oct 2013 00:07:00	1237173	15940664	hadcm3n_n65v_1880_40_008396317_1	25,920	29,386	1.1337