Task 15372496

Name	hadcm3n_zn4y_1880_40_008198494_3
Workunit	8353618
Created	20 Oct 2012, 23:39:20 UTC
Sent	20 Oct 2012, 23:39:28 UTC
Report deadline	20 Jan 2013, 7:06:39 UTC
Received	23 Dec 2012, 4:06:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1179495
Run time	14 days 19 hours 53 min 20 sec
CPU time	12 days 13 hours 18 min 8 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.55 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3160, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3228, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:52:44 (3028): No heartbeat from core client for 30 sec - exiting 22:52:45 (3028): No heartbeat from core client for 30 sec - exiting 22:52:46 (3028): No heartbeat from core client for 30 sec - exiting 22:52:47 (3028): No heartbeat from core client for 30 sec - exiting 22:52:48 (3028): No heartbeat from core client for 30 sec - exiting 22:52:49 (3028): No heartbeat from core client for 30 sec - exiting 22:52:50 (3028): No heartbeat from core client for 30 sec - exiting 22:52:51 (3028): No heartbeat from core client for 30 sec - exiting 22:52:52 (3028): No heartbeat from core client for 30 sec - exiting 22:52:53 (3028): No heartbeat from core client for 30 sec - exiting 22:52:54 (3028): No heartbeat from core client for 30 sec - exiting 22:52:55 (3028): No heartbeat from core client for 30 sec - exiting 22:52:56 (3028): No heartbeat from core client for 30 sec - exiting 22:52:58 (3028): No heartbeat from core client for 30 sec - exiting 22:52:59 (3028): No heartbeat from core client for 30 sec - exiting 22:53:00 (3028): No heartbeat from core client for 30 sec - exiting 22:53:01 (3028): No heartbeat from core client for 30 sec - exiting 22:53:02 (3028): No heartbeat from core client for 30 sec - exiting 22:53:03 (3028): No heartbeat from core client for 30 sec - exiting 22:53:04 (3028): No heartbeat from core client for 30 sec - exiting 22:53:05 (3028): No heartbeat from core client for 30 sec - exiting 22:53:06 (3028): No heartbeat from core client for 30 sec - exiting 22:53:07 (3028): No heartbeat from core client for 30 sec - exiting 22:53:08 (3028): No heartbeat from core client for 30 sec - exiting 22:53:10 (3028): No heartbeat from core client for 30 sec - exiting 22:53:11 (3028): No heartbeat from core client for 30 sec - exiting 22:53:12 (3028): No heartbeat from core client for 30 sec - exiting 22:53:13 (3028): No heartbeat from core client for 30 sec - exiting 22:53:14 (3028): No heartbeat from core client for 30 sec - exiting 22:53:15 (3028): No heartbeat from core client for 30 sec - exiting 22:53:16 (3028): No heartbeat from core client for 30 sec - exiting 22:53:17 (3028): No heartbeat from core client for 30 sec - exiting 22:53:18 (3028): No heartbeat from core client for 30 sec - exiting 22:53:19 (3028): No heartbeat from core client for 30 sec - exiting 22:53:20 (3028): No heartbeat from core client for 30 sec - exiting 22:53:22 (3028): No heartbeat from core client for 30 sec - exiting 22:53:23 (3028): No heartbeat from core client for 30 sec - exiting 22:53:24 (3028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:46 (2348): No heartbeat from core client for 30 sec - exiting 20:16:48 (2348): No heartbeat from core client for 30 sec - exiting 20:16:49 (2348): No heartbeat from core client for 30 sec - exiting 20:16:50 (2348): No heartbeat from core client for 30 sec - exiting 20:16:51 (2348): No heartbeat from core client for 30 sec - exiting 20:16:52 (2348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:48:35 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:51:17 (2092): No heartbeat from core client for 30 sec - exiting 07:51:18 (2092): No heartbeat from core client for 30 sec - exiting 07:51:20 (2092): No heartbeat from core client for 30 sec - exiting 07:51:21 (2092): No heartbeat from core client for 30 sec - exiting 07:51:22 (2092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:41:02 (5544): No heartbeat from core client for 30 sec - exiting 20:41:05 (5544): No heartbeat from core client for 30 sec - exiting 20:41:06 (5544): No heartbeat from core client for 30 sec - exiting 20:41:07 (5544): No heartbeat from core client for 30 sec - exiting 20:41:08 (5544): No heartbeat from core client for 30 sec - exiting 20:41:09 (5544): No heartbeat from core client for 30 sec - exiting 20:41:10 (5544): No heartbeat from core client for 30 sec - exiting 20:41:11 (5544): No heartbeat from core client for 30 sec - exiting 20:41:12 (5544): No heartbeat from core client for 30 sec - exiting 20:41:13 (5544): No heartbeat from core client for 30 sec - exiting 20:41:14 (5544): No heartbeat from core client for 30 sec - exiting 20:41:15 (5544): No heartbeat from core client for 30 sec - exiting 20:41:17 (5544): No heartbeat from core client for 30 sec - exiting 20:41:18 (5544): No heartbeat from core client for 30 sec - exiting 20:41:19 (5544): No heartbeat from core client for 30 sec - exiting 20:41:20 (5544): No heartbeat from core client for 30 sec - exiting 20:41:21 (5544): No heartbeat from core client for 30 sec - exiting 20:41:22 (5544): No heartbeat from core client for 30 sec - exiting 20:41:23 (5544): No heartbeat from core client for 30 sec - exiting 20:41:24 (5544): No heartbeat from core client for 30 sec - exiting 20:41:25 (5544): No heartbeat from core client for 30 sec - exiting 20:41:26 (5544): No heartbeat from core client for 30 sec - exiting 20:41:27 (5544): No heartbeat from core client for 30 sec - exiting 20:41:29 (5544): No heartbeat from core client for 30 sec - exiting 20:41:30 (5544): No heartbeat from core client for 30 sec - exiting 20:41:31 (5544): No heartbeat from core client for 30 sec - exiting 20:41:32 (5544): No heartbeat from core client for 30 sec - exiting 20:41:33 (5544): No heartbeat from core client for 30 sec - exiting 20:41:34 (5544): No heartbeat from core client for 30 sec - exiting 20:41:35 (5544): No heartbeat from core client for 30 sec - exiting 20:41:36 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:37 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:00:10 (3232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:40:34 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:30:00 (7040): No heartbeat from core client for 30 sec - exiting 17:30:02 (7040): No heartbeat from core client for 30 sec - exiting 17:30:03 (7040): No heartbeat from core client for 30 sec - exiting 17:30:04 (7040): No heartbeat from core client for 30 sec - exiting 17:30:05 (7040): No heartbeat from core client for 30 sec - exiting 17:30:06 (7040): No heartbeat from core client for 30 sec - exiting 17:30:07 (7040): No heartbeat from core client for 30 sec - exiting 17:30:08 (7040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:09 (7040): No heartbeat from core client for 30 sec - exiting 17:30:10 (7040): No heartbeat from core client for 30 sec - exiting 17:30:11 (7040): No heartbeat from core client for 30 sec - exiting 17:30:12 (7040): No heartbeat from core client for 30 sec - exiting 17:30:14 (7040): No heartbeat from core client for 30 sec - exiting 17:30:15 (7040): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:15:18 (8808): No heartbeat from core client for 30 sec - exiting 19:15:19 (8808): No heartbeat from core client for 30 sec - exiting 19:15:20 (8808): No heartbeat from core client for 30 sec - exiting 19:15:21 (8808): No heartbeat from core client for 30 sec - exiting 19:15:23 (8808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:15:24 (8808): No heartbeat from core client for 30 sec - exiting 19:17:15 (8816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:09:13 (4224): No heartbeat from core client for 30 sec - exiting 04:09:15 (4224): No heartbeat from core client for 30 sec - exiting 04:09:16 (4224): No heartbeat from core client for 30 sec - exiting 04:09:17 (4224): No heartbeat from core client for 30 sec - exiting 04:09:18 (4224): No heartbeat from core client for 30 sec - exiting 04:09:19 (4224): No heartbeat from core client for 30 sec - exiting 04:09:20 (4224): No heartbeat from core client for 30 sec - exiting 04:09:21 (4224): No heartbeat from core client for 30 sec - exiting 04:09:22 (4224): No heartbeat from core client for 30 sec - exiting 04:09:23 (4224): No heartbeat from core client for 30 sec - exiting 04:09:24 (4224): No heartbeat from core client for 30 sec - exiting 04:09:25 (4224): No heartbeat from core client for 30 sec - exiting 04:09:26 (4224): No heartbeat from core client for 30 sec - exiting 04:09:27 (4224): No heartbeat from core client for 30 sec - exiting 04:09:29 (4224): No heartbeat from core client for 30 sec - exiting 04:09:30 (4224): No heartbeat from core client for 30 sec - exiting 04:09:31 (4224): No heartbeat from core client for 30 sec - exiting 04:09:32 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2108, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5092, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FA3AB3 read attempt to address 0x4039E39B Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FA7373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zn4y_1880_40_008198494/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FA7373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zn4y_1880_40_008198494/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Dec 2012 17:46:29	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	518,400	1,073,283	2.0704
21 Dec 2012 08:11:34	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	492,480	1,016,733	2.0645
18 Dec 2012 18:53:15	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	466,560	959,599	2.0568
15 Dec 2012 21:21:41	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	440,640	908,419	2.0616
14 Dec 2012 20:24:49	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	414,720	859,870	2.0734
14 Dec 2012 20:24:49	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	388,800	808,889	2.0805
07 Dec 2012 22:34:11	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	362,880	753,000	2.0751
03 Dec 2012 21:39:17	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	336,960	698,185	2.0720
01 Dec 2012 04:35:22	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	311,040	642,977	2.0672
27 Nov 2012 18:05:57	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	285,120	585,429	2.0533
19 Nov 2012 20:43:56	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	259,200	532,496	2.0544
18 Nov 2012 10:45:06	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	233,280	477,761	2.0480
16 Nov 2012 03:10:07	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	207,360	424,963	2.0494
11 Nov 2012 20:45:05	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	181,440	373,587	2.0590
08 Nov 2012 06:14:16	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	155,520	321,086	2.0646
06 Nov 2012 21:45:52	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	129,600	268,770	2.0738
05 Nov 2012 04:07:29	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	103,680	216,586	2.0890
04 Nov 2012 10:40:54	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	77,760	161,058	2.0712
30 Oct 2012 21:27:15	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	51,840	105,685	2.0387
25 Oct 2012 21:16:51	1179495	15372496	hadcm3n_zn4y_1880_40_008198494_3	25,920	53,045	2.0465