Task 15878642

Name	hadcm3n_zhl4_1920_40_008315744_4
Workunit	8466879
Created	3 Jul 2013, 9:34:07 UTC
Sent	3 Jul 2013, 9:48:57 UTC
Report deadline	2 Oct 2013, 17:16:08 UTC
Received	14 Aug 2013, 20:48:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1129647
Run time	16 days 19 hours 21 min 24 sec
CPU time	15 days 20 hours 53 min 2 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:56:28 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:53:28 (5624): No heartbeat from core client for 30 sec - exiting 05:53:29 (5624): No heartbeat from core client for 30 sec - exiting 05:53:30 (5624): No heartbeat from core client for 30 sec - exiting 05:53:31 (5624): No heartbeat from core client for 30 sec - exiting 05:53:32 (5624): No heartbeat from core client for 30 sec - exiting 05:53:33 (5624): No heartbeat from core client for 30 sec - exiting 05:53:34 (5624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:19 (5544): No heartbeat from core client for 30 sec - exiting 17:08:21 (5544): No heartbeat from core client for 30 sec - exiting 17:08:22 (5544): No heartbeat from core client for 30 sec - exiting 17:08:23 (5544): No heartbeat from core client for 30 sec - exiting 17:08:24 (5544): No heartbeat from core client for 30 sec - exiting 17:08:25 (5544): No heartbeat from core client for 30 sec - exiting 17:08:26 (5544): No heartbeat from core client for 30 sec - exiting 17:08:27 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:55 (5596): No heartbeat from core client for 30 sec - exiting 05:35:57 (5596): No heartbeat from core client for 30 sec - exiting 05:35:58 (5596): No heartbeat from core client for 30 sec - exiting 05:35:59 (5596): No heartbeat from core client for 30 sec - exiting 05:36:00 (5596): No heartbeat from core client for 30 sec - exiting 05:36:01 (5596): No heartbeat from core client for 30 sec - exiting 05:36:02 (5596): No heartbeat from core client for 30 sec - exiting 05:36:03 (5596): No heartbeat from core client for 30 sec - exiting 05:36:04 (5596): No heartbeat from core client for 30 sec - exiting 05:36:05 (5596): No heartbeat from core client for 30 sec - exiting 05:36:06 (5596): No heartbeat from core client for 30 sec - exiting 05:36:07 (5596): No heartbeat from core client for 30 sec - exiting 05:36:08 (5596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:30:41 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5596, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:53:40 (5976): No heartbeat from core client for 30 sec - exiting 16:53:41 (5976): No heartbeat from core client for 30 sec - exiting 16:53:42 (5976): No heartbeat from core client for 30 sec - exiting 16:53:43 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:42:29 (5636): No heartbeat from core client for 30 sec - exiting 05:42:31 (5636): No heartbeat from core client for 30 sec - exiting 05:42:32 (5636): No heartbeat from core client for 30 sec - exiting 05:42:33 (5636): No heartbeat from core client for 30 sec - exiting 05:42:34 (5636): No heartbeat from core client for 30 sec - exiting 05:42:35 (5636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:38 (5292): No heartbeat from core client for 30 sec - exiting 16:43:39 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:37:42 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:47:16 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4676, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:13:14 (9600): No heartbeat from core client for 30 sec - exiting 16:13:15 (9600): No heartbeat from core client for 30 sec - exiting 16:13:16 (9600): No heartbeat from core client for 30 sec - exiting 16:13:17 (9600): No heartbeat from core client for 30 sec - exiting 16:13:18 (9600): No heartbeat from core client for 30 sec - exiting 16:13:19 (9600): No heartbeat from core client for 30 sec - exiting 16:13:20 (9600): No heartbeat from core client for 30 sec - exiting 16:13:21 (9600): No heartbeat from core client for 30 sec - exiting 16:13:22 (9600): No heartbeat from core client for 30 sec - exiting 16:13:23 (9600): No heartbeat from core client for 30 sec - exiting 16:13:24 (9600): No heartbeat from core client for 30 sec - exiting 16:13:25 (9600): No heartbeat from core client for 30 sec - exiting 16:13:26 (9600): No heartbeat from core client for 30 sec - exiting 16:13:27 (9600): No heartbeat from core client for 30 sec - exiting 16:13:28 (9600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:44:37 (3904): No heartbeat from core client for 30 sec - exiting 16:44:38 (3904): No heartbeat from core client for 30 sec - exiting 16:44:39 (3904): No heartbeat from core client for 30 sec - exiting 16:44:40 (3904): No heartbeat from core client for 30 sec - exiting 16:44:41 (3904): No heartbeat from core client for 30 sec - exiting 16:44:42 (3904): No heartbeat from core client for 30 sec - exiting 16:44:43 (3904): No heartbeat from core client for 30 sec - exiting 16:44:44 (3904): No heartbeat from core client for 30 sec - exiting 16:44:45 (3904): No heartbeat from core client for 30 sec - exiting 16:44:46 (3904): No heartbeat from core client for 30 sec - exiting 16:44:47 (3904): No heartbeat from core client for 30 sec - exiting 16:44:48 (3904): No heartbeat from core client for 30 sec - exiting 16:44:49 (3904): No heartbeat from core client for 30 sec - exiting 16:44:50 (3904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:44:51 (3904): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:43:58 (4508): No heartbeat from core client for 30 sec - exiting 16:43:59 (4508): No heartbeat from core client for 30 sec - exiting 16:44:00 (4508): No heartbeat from core client for 30 sec - exiting 16:44:01 (4508): No heartbeat from core client for 30 sec - exiting 16:44:02 (4508): No heartbeat from core client for 30 sec - exiting 16:44:03 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:37:19 (5272): No heartbeat from core client for 30 sec - exiting 16:37:20 (5272): No heartbeat from core client for 30 sec - exiting 16:37:21 (5272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 20:50:13	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	777,600	1,371,178	1.7633
14 Aug 2013 20:50:13	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	751,680	1,326,548	1.7648
14 Aug 2013 20:50:13	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	725,760	1,282,166	1.7667
14 Aug 2013 20:50:13	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	699,840	1,238,468	1.7696
14 Aug 2013 20:50:13	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	673,920	1,194,447	1.7724
14 Aug 2013 20:50:12	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	648,000	1,150,200	1.7750
14 Aug 2013 20:50:12	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	622,080	1,106,747	1.7791
14 Aug 2013 20:50:12	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	596,160	1,061,783	1.7810
14 Aug 2013 20:50:12	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	570,240	1,016,150	1.7820
30 Jul 2013 10:21:58	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	544,320	970,277	1.7825
30 Jul 2013 10:21:58	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	518,400	923,967	1.7823
25 Jul 2013 05:43:45	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	492,480	877,143	1.7811
24 Jul 2013 04:47:48	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	466,560	830,835	1.7808
23 Jul 2013 22:01:46	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	440,640	784,141	1.7796
23 Jul 2013 21:22:49	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	414,720	738,565	1.7809
23 Jul 2013 20:38:28	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	388,800	693,122	1.7827
23 Jul 2013 19:56:12	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	362,880	647,101	1.7832
23 Jul 2013 18:58:54	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	336,960	600,892	1.7833
23 Jul 2013 14:27:00	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	311,040	554,338	1.7822
23 Jul 2013 14:26:59	1129647	15878642	hadcm3n_zhl4_1920_40_008315744_4	285,120	508,137	1.7822