Task 15689811

Name	hadcm3n_u78e_2020_40_008339305_0
Workunit	8490166
Created	28 Mar 2013, 17:53:13 UTC
Sent	28 Mar 2013, 17:58:17 UTC
Report deadline	28 Jun 2013, 1:25:28 UTC
Received	17 Apr 2013, 20:18:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1241003
Run time	6 days 0 hours 17 min 52 sec
CPU time	4 days 18 hours 6 min 45 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	3.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> Enheden genkender ikke kommandoen. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 20:51:14 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:56:00 (5616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:10:31 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:48 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:35 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:54:54 (4848): No heartbeat from core client for 30 sec - exiting 18:54:56 (4848): No heartbeat from core client for 30 sec - exiting 18:54:57 (4848): No heartbeat from core client for 30 sec - exiting 18:54:58 (4848): No heartbeat from core client for 30 sec - exiting 18:54:59 (4848): No heartbeat from core client for 30 sec - exiting 18:55:00 (4848): No heartbeat from core client for 30 sec - exiting 18:55:01 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:16:25 (5192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:21 (3248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:06 (5020): No heartbeat from core client for 30 sec - exiting 17:03:07 (5020): No heartbeat from core client for 30 sec - exiting 17:03:08 (5020): No heartbeat from core client for 30 sec - exiting 17:03:09 (5020): No heartbeat from core client for 30 sec - exiting 17:03:11 (5020): No heartbeat from core client for 30 sec - exiting 17:03:12 (5020): No heartbeat from core client for 30 sec - exiting 17:03:13 (5020): No heartbeat from core client for 30 sec - exiting 17:03:14 (5020): No heartbeat from core client for 30 sec - exiting 17:03:15 (5020): No heartbeat from core client for 30 sec - exiting 17:03:16 (5020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:23:50 (4980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:01 (5416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:02 (5416): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:05:51 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:42:53 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:54 (4284): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:53:36 (5472): No heartbeat from core client for 30 sec - exiting 08:53:37 (5472): No heartbeat from core client for 30 sec - exiting 08:53:38 (5472): No heartbeat from core client for 30 sec - exiting 08:53:39 (5472): No heartbeat from core client for 30 sec - exiting 08:53:40 (5472): No heartbeat from core client for 30 sec - exiting 08:53:41 (5472): No heartbeat from core client for 30 sec - exiting 08:53:42 (5472): No heartbeat from core client for 30 sec - exiting 08:53:43 (5472): No heartbeat from core client for 30 sec - exiting 08:53:44 (5472): No heartbeat from core client for 30 sec - exiting 08:53:45 (5472): No heartbeat from core client for 30 sec - exiting 08:53:46 (5472): No heartbeat from core client for 30 sec - exiting 08:53:47 (5472): No heartbeat from core client for 30 sec - exiting 08:53:48 (5472): No heartbeat from core client for 30 sec - exiting 08:53:49 (5472): No heartbeat from core client for 30 sec - exiting 08:53:50 (5472): No heartbeat from core client for 30 sec - exiting 08:53:51 (5472): No heartbeat from core client for 30 sec - exiting 08:53:52 (5472): No heartbeat from core client for 30 sec - exiting 08:53:53 (5472): No heartbeat from core client for 30 sec - exiting 08:53:54 (5472): No heartbeat from core client for 30 sec - exiting 08:53:55 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:13 (4736): No heartbeat from core client for 30 sec - exiting 12:16:14 (4736): No heartbeat from core client for 30 sec - exiting 12:16:15 (4736): No heartbeat from core client for 30 sec - exiting 12:16:16 (4736): No heartbeat from core client for 30 sec - exiting 12:16:17 (4736): No heartbeat from core client for 30 sec - exiting 12:16:18 (4736): No heartbeat from core client for 30 sec - exiting 12:16:19 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:09:08 (1664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:03 (264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:21 (4360): No heartbeat from core client for 30 sec - exiting 08:52:22 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:07:33 (5360): No heartbeat from core client for 30 sec - exiting 09:07:34 (5360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:35 (5360): No heartbeat from core client for 30 sec - exiting 09:08:11 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:19 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:28:39 (264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:01 (2168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:47 (5260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:53:26 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:27 (5356): No heartbeat from core client for 30 sec - exiting 10:56:31 (5548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:32:31 (5072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:49:07 (984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:43 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:44 (4952): No heartbeat from core client for 30 sec - exiting 08:52:45 (4952): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:12:57 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:55 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:56 (3972): No heartbeat from core client for 30 sec - exiting 08:13:57 (3972): No heartbeat from core client for 30 sec - exiting 08:13:58 (3972): No heartbeat from core client for 30 sec - exiting 08:13:59 (3972): No heartbeat from core client for 30 sec - exiting 08:14:00 (3972): No heartbeat from core client for 30 sec - exiting 08:14:01 (3972): No heartbeat from core client for 30 sec - exiting 08:14:02 (3972): No heartbeat from core client for 30 sec - exiting 08:14:03 (3972): No heartbeat from core client for 30 sec - exiting 08:14:04 (3972): No heartbeat from core client for 30 sec - exiting 08:14:05 (3972): No heartbeat from core client for 30 sec - exiting 08:17:06 (800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:49:35 (2996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:22 (4720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:42 (4688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:58:19 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:16 (2888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:03 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:43 (1076): No heartbeat from core client for 30 sec - exiting 08:20:44 (1076): No heartbeat from core client for 30 sec - exiting 08:20:45 (1076): No heartbeat from core client for 30 sec - exiting 08:20:46 (1076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:22:17 (1188): No heartbeat from core client for 30 sec - exiting 08:22:18 (1188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:22:19 (1188): No heartbeat from core client for 30 sec - exiting 08:22:55 (2840): No heartbeat from core client for 30 sec - exiting 08:22:56 (2840): No heartbeat from core client for 30 sec - exiting 08:22:57 (2840): No heartbeat from core client for 30 sec - exiting 08:22:58 (2840): No heartbeat from core client for 30 sec - exiting 08:22:59 (2840): No heartbeat from core client for 30 sec - exiting 08:23:00 (2840): No heartbeat from core client for 30 sec - exiting 08:23:01 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 01:49:18 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:59:53 (5340): No heartbeat from core client for 30 sec - exiting 03:59:54 (5340): No heartbeat from core client for 30 sec - exiting 03:59:55 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:54 (1892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:03 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:19 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:23:56 (2848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:13 (4140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:39:04 (1440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:05 (1440): No heartbeat from core client for 30 sec - exiting 07:39:06 (1440): No heartbeat from core client for 30 sec - exiting 07:39:07 (1440): No heartbeat from core client for 30 sec - exiting 07:39:08 (1440): No heartbeat from core client for 30 sec - exiting 07:39:09 (1440): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Apr 2013 16:36:15	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	414,720	407,103	0.9816
14 Apr 2013 22:47:09	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	388,800	386,821	0.9949
13 Apr 2013 16:26:18	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	362,880	366,149	1.0090
12 Apr 2013 08:49:21	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	336,960	340,745	1.0112
10 Apr 2013 19:53:15	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	311,040	315,005	1.0127
09 Apr 2013 18:50:05	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	285,120	288,827	1.0130
08 Apr 2013 12:38:30	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	259,200	262,623	1.0132
07 Apr 2013 08:07:41	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	233,280	236,013	1.0117
06 Apr 2013 12:14:55	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	207,360	209,740	1.0115
05 Apr 2013 18:48:35	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	181,440	183,548	1.0116
04 Apr 2013 10:46:24	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	155,520	157,601	1.0134
03 Apr 2013 13:12:20	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	129,600	131,126	1.0118
01 Apr 2013 18:40:06	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	103,680	104,560	1.0085
01 Apr 2013 09:34:41	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	77,760	78,596	1.0108
31 Mar 2013 09:09:14	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	51,840	52,602	1.0147
29 Mar 2013 20:59:00	1241003	15689811	hadcm3n_u78e_2020_40_008339305_0	25,920	26,419	1.0193