Task 15419993

Name	hadcm3n_zi89_1880_40_008242722_2
Workunit	8397846
Created	30 Oct 2012, 14:58:52 UTC
Sent	30 Oct 2012, 14:58:59 UTC
Report deadline	29 Jan 2013, 22:26:10 UTC
Received	11 Nov 2012, 4:52:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1206793
Run time	6 days 17 hours 9 min 56 sec
CPU time	5 days 6 hours 20 min 8 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3188, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:49:21 (5688): No heartbeat from core client for 30 sec - exiting 08:49:22 (5688): No heartbeat from core client for 30 sec - exiting 08:49:23 (5688): No heartbeat from core client for 30 sec - exiting 08:49:24 (5688): No heartbeat from core client for 30 sec - exiting 08:49:25 (5688): No heartbeat from core client for 30 sec - exiting 08:49:26 (5688): No heartbeat from core client for 30 sec - exiting 08:49:27 (5688): No heartbeat from core client for 30 sec - exiting 08:49:28 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:26 (6864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:30:43 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:28 (1768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:34 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:14 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:39:41 (2096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:00 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:02:28 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:46:57 (5508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:38:34 (4256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:24 (160): No heartbeat from core client for 30 sec - exiting 12:35:25 (160): No heartbeat from core client for 30 sec - exiting 12:35:26 (160): No heartbeat from core client for 30 sec - exiting 12:35:27 (160): No heartbeat from core client for 30 sec - exiting 12:35:28 (160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:21:27 (7020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:03:06 (7096): No heartbeat from core client for 30 sec - exiting 22:03:07 (7096): No heartbeat from core client for 30 sec - exiting 22:03:08 (7096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:03:09 (7096): No heartbeat from core client for 30 sec - exiting 09:13:16 (5260): No heartbeat from core client for 30 sec - exiting 09:13:17 (5260): No heartbeat from core client for 30 sec - exiting 09:13:18 (5260): No heartbeat from core client for 30 sec - exiting 09:13:19 (5260): No heartbeat from core client for 30 sec - exiting 09:13:20 (5260): No heartbeat from core client for 30 sec - exiting 09:13:21 (5260): No heartbeat from core client for 30 sec - exiting 09:13:22 (5260): No heartbeat from core client for 30 sec - exiting 09:13:23 (5260): No heartbeat from core client for 30 sec - exiting 09:13:24 (5260): No heartbeat from core client for 30 sec - exiting 09:13:25 (5260): No heartbeat from core client for 30 sec - exiting 09:13:26 (5260): No heartbeat from core client for 30 sec - exiting 09:13:27 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:09:18 (1544): No heartbeat from core client for 30 sec - exiting 08:09:20 (1544): No heartbeat from core client for 30 sec - exiting 08:09:21 (1544): No heartbeat from core client for 30 sec - exiting 08:09:22 (1544): No heartbeat from core client for 30 sec - exiting 08:09:23 (1544): No heartbeat from core client for 30 sec - exiting 08:09:24 (1544): No heartbeat from core client for 30 sec - exiting 08:09:25 (1544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:30:40 (3332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:00 (6156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:08:48 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:46 (1980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:20 (6712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:15 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1132, iMonCtr=1 Model crash detected, will try to restart... 05:36:51 (6060): No heartbeat from core client for 30 sec - exiting 05:36:52 (6060): No heartbeat from core client for 30 sec - exiting 05:36:53 (6060): No heartbeat from core client for 30 sec - exiting 05:36:54 (6060): No heartbeat from core client for 30 sec - exiting 05:36:55 (6060): No heartbeat from core client for 30 sec - exiting 05:36:56 (6060): No heartbeat from core client for 30 sec - exiting 05:36:57 (6060): No heartbeat from core client for 30 sec - exiting 05:36:58 (6060): No heartbeat from core client for 30 sec - exiting 05:36:59 (6060): No heartbeat from core client for 30 sec - exiting 05:37:00 (6060): No heartbeat from core client for 30 sec - exiting 05:37:01 (6060): No heartbeat from core client for 30 sec - exiting 05:37:02 (6060): No heartbeat from core client for 30 sec - exiting 05:37:03 (6060): No heartbeat from core client for 30 sec - exiting 05:37:04 (6060): No heartbeat from core client for 30 sec - exiting 05:37:05 (6060): No heartbeat from core client for 30 sec - exiting 05:37:06 (6060): No heartbeat from core client for 30 sec - exiting 05:37:07 (6060): No heartbeat from core client for 30 sec - exiting 05:37:08 (6060): No heartbeat from core client for 30 sec - exiting 05:37:09 (6060): No heartbeat from core client for 30 sec - exiting 05:37:10 (6060): No heartbeat from core client for 30 sec - exiting 05:37:11 (6060): No heartbeat from core client for 30 sec - exiting 05:37:12 (6060): No heartbeat from core client for 30 sec - exiting 05:37:13 (6060): No heartbeat from core client for 30 sec - exiting 05:37:14 (6060): No heartbeat from core client for 30 sec - exiting 05:37:15 (6060): No heartbeat from core client for 30 sec - exiting 05:37:16 (6060): No heartbeat from core client for 30 sec - exiting 05:37:17 (6060): No heartbeat from core client for 30 sec - exiting 05:37:18 (6060): No heartbeat from core client for 30 sec - exiting 05:37:19 (6060): No heartbeat from core client for 30 sec - exiting 05:37:20 (6060): No heartbeat from core client for 30 sec - exiting 05:37:21 (6060): No heartbeat from core client for 30 sec - exiting 05:37:22 (6060): No heartbeat from core client for 30 sec - exiting 05:37:23 (6060): No heartbeat from core client for 30 sec - exiting 05:37:24 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:53 (3104): No heartbeat from core client for 30 sec - exiting 05:36:54 (3104): No heartbeat from core client for 30 sec - exiting 05:36:55 (3104): No heartbeat from core client for 30 sec - exiting 05:36:56 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:40:49 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:54 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:55 (4652): No heartbeat from core client for 30 sec - exiting 05:43:56 (4652): No heartbeat from core client for 30 sec - exiting 05:43:57 (4652): No heartbeat from core client for 30 sec - exiting 05:43:58 (4652): No heartbeat from core client for 30 sec - exiting 05:43:59 (4652): No heartbeat from core client for 30 sec - exiting 05:44:00 (4652): No heartbeat from core client for 30 sec - exiting 05:44:01 (4652): No heartbeat from core client for 30 sec - exiting 05:44:02 (4652): No heartbeat from core client for 30 sec - exiting 05:44:03 (4652): No heartbeat from core client for 30 sec - exiting 05:44:04 (4652): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 06:03:32 (5000): No heartbeat from core client for 30 sec - exiting 06:03:33 (5000): No heartbeat from core client for 30 sec - exiting 06:03:34 (5000): No heartbeat from core client for 30 sec - exiting 06:03:35 (5000): No heartbeat from core client for 30 sec - exiting 06:03:36 (5000): No heartbeat from core client for 30 sec - exiting 06:03:37 (5000): No heartbeat from core client for 30 sec - exiting 06:03:38 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:34 (3832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:12 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:26 (1500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:27 (1500): No heartbeat from core client for 30 sec - exiting 15:01:28 (1500): No heartbeat from core client for 30 sec - exiting 15:01:29 (1500): No heartbeat from core client for 30 sec - exiting 15:01:30 (1500): No heartbeat from core client for 30 sec - exiting 15:01:31 (1500): No heartbeat from core client for 30 sec - exiting 15:01:32 (1500): No heartbeat from core client for 30 sec - exiting 15:01:33 (1500): No heartbeat from core client for 30 sec - exiting 15:01:34 (1500): No heartbeat from core client for 30 sec - exiting 15:01:35 (1500): No heartbeat from core client for 30 sec - exiting 15:01:36 (1500): No heartbeat from core client for 30 sec - exiting 18:25:06 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:26:00 (2244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:26:56 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:33 (4340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:37:14 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:42:14 (4292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:08 (2036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:54 (4160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:54 (5112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:01 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:05 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:02:52 (3668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:26 (6816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:40 (6284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:22:22 (6300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:14 (6828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:11 (1552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:00 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:39:54 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:24 (4472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:17 (6392): No heartbeat from core client for 30 sec - exiting 10:18:18 (6392): No heartbeat from core client for 30 sec - exiting 10:18:19 (6392): No heartbeat from core client for 30 sec - exiting 10:18:20 (6392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:21 (6392): No heartbeat from core client for 30 sec - exiting 10:20:41 (6448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:44 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:46 (7152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:43 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:11 (6924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:03:15 (3468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:11 (7060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:06 (628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:54 (6512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:04 (7008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:41 (4628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:55 (6520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:14:50 (1788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:14:51 (1788): No heartbeat from core client for 30 sec - exiting 13:14:52 (1788): No heartbeat from core client for 30 sec - exiting 13:14:53 (1788): No heartbeat from core client for 30 sec - exiting 13:14:54 (1788): No heartbeat from core client for 30 sec - exiting 13:14:55 (1788): No heartbeat from core client for 30 sec - exiting 13:14:56 (1788): No heartbeat from core client for 30 sec - exiting 13:14:57 (1788): No heartbeat from core client for 30 sec - exiting 13:14:58 (1788): No heartbeat from core client for 30 sec - exiting 13:14:59 (1788): No heartbeat from core client for 30 sec - exiting 13:15:00 (1788): No heartbeat from core client for 30 sec - exiting 13:19:14 (6416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:19:15 (6416): No heartbeat from core client for 30 sec - exiting 13:19:16 (6416): No heartbeat from core client for 30 sec - exiting 13:19:17 (6416): No heartbeat from core client for 30 sec - exiting 13:19:18 (6416): No heartbeat from core client for 30 sec - exiting 13:19:19 (6416): No heartbeat from core client for 30 sec - exiting 13:20:03 (6740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:22:10 (6676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:22:11 (6676): No heartbeat from core client for 30 sec - exiting 13:22:12 (6676): No heartbeat from core client for 30 sec - exiting 13:22:13 (6676): No heartbeat from core client for 30 sec - exiting 13:22:14 (6676): No heartbeat from core client for 30 sec - exiting 13:22:15 (6676): No heartbeat from core client for 30 sec - exiting 13:22:18 (6676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:50:35 (7136): No heartbeat from core client for 30 sec - exiting 20:50:36 (7136): No heartbeat from core client for 30 sec - exiting 20:50:37 (7136): No heartbeat from core client for 30 sec - exiting 20:50:38 (7136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Nov 2012 23:56:31	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	362,880	443,294	1.2216
10 Nov 2012 03:40:13	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	336,960	414,430	1.2299
09 Nov 2012 17:35:08	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	311,040	385,116	1.2382
09 Nov 2012 01:39:25	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	285,120	354,982	1.2450
08 Nov 2012 04:49:05	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	259,200	324,323	1.2512
07 Nov 2012 17:03:53	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	233,280	289,901	1.2427
06 Nov 2012 13:41:19	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	207,360	255,105	1.2303
06 Nov 2012 02:44:53	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	181,440	221,829	1.2226
05 Nov 2012 15:24:13	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	155,520	190,499	1.2249
04 Nov 2012 21:20:50	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	129,600	159,026	1.2271
03 Nov 2012 22:08:53	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	103,680	126,877	1.2237
03 Nov 2012 00:33:39	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	77,760	94,192	1.2113
02 Nov 2012 00:53:43	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	51,840	63,244	1.2200
31 Oct 2012 22:59:31	1206793	15419993	hadcm3n_zi89_1880_40_008242722_2	25,920	32,207	1.2426