Task 13641224

Name	hadcm3n_yaxf_1900_40_007524313_3
Workunit	7721788
Created	17 Nov 2011, 5:58:59 UTC
Sent	18 Nov 2011, 9:23:04 UTC
Report deadline	17 Feb 2012, 16:50:15 UTC
Received	9 Dec 2011, 11:26:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1180204
Run time	8 days 0 hours 49 min 15 sec
CPU time	5 days 11 hours 25 min 18 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.84 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5508, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:30:35 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3204, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 21:25:39 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:54 (1932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:25 (2208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:11:06 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:16:08 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:38 (3632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:04:11 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:13 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:18:02 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:16:17 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=728, iMonCtr=1 Model crash detected, will try to restart... 20:16:40 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:41 (4560): No heartbeat from core client for 30 sec - exiting 21:25:22 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:26:41 (1084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:22:52 (3856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:30 (6036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:48:27 (1708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:28 (1708): No heartbeat from core client for 30 sec - exiting 10:48:29 (1708): No heartbeat from core client for 30 sec - exiting 10:48:30 (1708): No heartbeat from core client for 30 sec - exiting 10:48:31 (1708): No heartbeat from core client for 30 sec - exiting 10:48:32 (1708): No heartbeat from core client for 30 sec - exiting 10:48:33 (1708): No heartbeat from core client for 30 sec - exiting 10:48:34 (1708): No heartbeat from core client for 30 sec - exiting 10:48:35 (1708): No heartbeat from core client for 30 sec - exiting 11:19:34 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:51:23 (1164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:51:24 (1164): No heartbeat from core client for 30 sec - exiting 13:51:26 (1164): No heartbeat from core client for 30 sec - exiting 13:51:27 (1164): No heartbeat from core client for 30 sec - exiting 13:51:28 (1164): No heartbeat from core client for 30 sec - exiting 13:51:29 (1164): No heartbeat from core client for 30 sec - exiting 13:51:30 (1164): No heartbeat from core client for 30 sec - exiting 13:51:31 (1164): No heartbeat from core client for 30 sec - exiting 13:51:32 (1164): No heartbeat from core client for 30 sec - exiting 13:51:33 (1164): No heartbeat from core client for 30 sec - exiting 13:53:24 (5068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:16:33 (5036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:41 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:44 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:57:45 (876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:54 (3032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:53 (5496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:54:09 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:52 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:03 (4628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:34 (1984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:41 (2908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:02:37 (684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:02:39 (684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:35:17 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:30:05 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5644, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:10:27 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:28 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:24:15 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 18:59:25 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:13:07 (4888): No heartbeat from core client for 30 sec - exiting 17:13:08 (4888): No heartbeat from core client for 30 sec - exiting 17:13:09 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:27 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:00:58 (3356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:59:51 (5380): No heartbeat from core client for 30 sec - exiting 16:59:52 (5380): No heartbeat from core client for 30 sec - exiting 16:59:53 (5380): No heartbeat from core client for 30 sec - exiting 16:59:54 (5380): No heartbeat from core client for 30 sec - exiting 16:59:55 (5380): No heartbeat from core client for 30 sec - exiting 16:59:56 (5380): No heartbeat from core client for 30 sec - exiting 16:59:57 (5380): No heartbeat from core client for 30 sec - exiting 16:59:58 (5380): No heartbeat from core client for 30 sec - exiting 16:59:59 (5380): No heartbeat from core client for 30 sec - exiting 17:00:00 (5380): No heartbeat from core client for 30 sec - exiting 17:00:01 (5380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:10 (5920): No heartbeat from core client for 30 sec - exiting 13:12:11 (5920): No heartbeat from core client for 30 sec - exiting 13:12:12 (5920): No heartbeat from core client for 30 sec - exiting 13:12:14 (5920): No heartbeat from core client for 30 sec - exiting 13:12:15 (5920): No heartbeat from core client for 30 sec - exiting 13:12:16 (5920): No heartbeat from core client for 30 sec - exiting 13:12:17 (5920): No heartbeat from core client for 30 sec - exiting 13:12:18 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:19 (5920): No heartbeat from core client for 30 sec - exiting 13:12:20 (5920): No heartbeat from core client for 30 sec - exiting 13:12:21 (5920): No heartbeat from core client for 30 sec - exiting 13:12:22 (5920): No heartbeat from core client for 30 sec - exiting 13:12:23 (5920): No heartbeat from core client for 30 sec - exiting 13:12:24 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:54:28 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:07:37 (1152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:40 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:45:17 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:14:00 (5648): No heartbeat from core client for 30 sec - exiting 15:14:01 (5648): No heartbeat from core client for 30 sec - exiting 15:14:02 (5648): No heartbeat from core client for 30 sec - exiting 15:14:03 (5648): No heartbeat from core client for 30 sec - exiting 15:14:04 (5648): No heartbeat from core client for 30 sec - exiting 15:14:05 (5648): No heartbeat from core client for 30 sec - exiting 15:14:06 (5648): No heartbeat from core client for 30 sec - exiting 15:14:07 (5648): No heartbeat from core client for 30 sec - exiting 15:14:08 (5648): No heartbeat from core client for 30 sec - exiting 15:14:09 (5648): No heartbeat from core client for 30 sec - exiting 15:14:10 (5648): No heartbeat from core client for 30 sec - exiting 15:14:11 (5648): No heartbeat from core client for 30 sec - exiting 15:14:12 (5648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Dec 2011 20:43:41	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	259,200	473,116	1.8253
06 Dec 2011 13:36:20	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	233,280	424,985	1.8218
05 Dec 2011 02:24:27	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	207,360	377,382	1.8199
04 Dec 2011 02:08:23	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	181,440	331,003	1.8243
03 Dec 2011 04:40:21	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	155,520	284,050	1.8265
01 Dec 2011 04:59:28	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	129,600	237,486	1.8325
28 Nov 2011 17:32:58	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	103,680	191,126	1.8434
23 Nov 2011 06:17:06	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	77,760	143,950	1.8512
21 Nov 2011 08:48:28	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	51,840	95,531	1.8428
20 Nov 2011 01:46:17	1180204	13641224	hadcm3n_yaxf_1900_40_007524313_3	25,920	48,370	1.8661