Task 15771139

Name	hadcm3n_n0eu_1920_40_008362365_0
Workunit	8513224
Created	10 May 2013, 11:42:47 UTC
Sent	10 May 2013, 11:42:57 UTC
Report deadline	9 Aug 2013, 19:10:08 UTC
Received	3 Jun 2013, 7:27:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1245161
Run time	6 days 0 hours 24 min 32 sec
CPU time	5 days 22 hours 48 min 26 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2840, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5148, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3960, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4336, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 10:02:20 (6308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:48 (2236): No heartbeat from core client for 30 sec - exiting 19:39:49 (2236): No heartbeat from core client for 30 sec - exiting 19:39:50 (2236): No heartbeat from core client for 30 sec - exiting 19:39:51 (2236): No heartbeat from core client for 30 sec - exiting 19:39:52 (2236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:02:21 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:00:31 (5332): No heartbeat from core client for 30 sec - exiting 10:00:33 (5332): No heartbeat from core client for 30 sec - exiting 10:00:34 (5332): No heartbeat from core client for 30 sec - exiting 10:00:35 (5332): No heartbeat from core client for 30 sec - exiting 10:00:36 (5332): No heartbeat from core client for 30 sec - exiting 10:00:37 (5332): No heartbeat from core client for 30 sec - exiting 10:00:38 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:26:23 (3480): No heartbeat from core client for 30 sec - exiting 18:26:25 (3480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:49:29 (3812): No heartbeat from core client for 30 sec - exiting 09:49:30 (3812): No heartbeat from core client for 30 sec - exiting 09:49:31 (3812): No heartbeat from core client for 30 sec - exiting 09:49:32 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:39:27 (4592): No heartbeat from core client for 30 sec - exiting 16:39:28 (4592): No heartbeat from core client for 30 sec - exiting 16:39:29 (4592): No heartbeat from core client for 30 sec - exiting 16:39:30 (4592): No heartbeat from core client for 30 sec - exiting 16:39:31 (4592): No heartbeat from core client for 30 sec - exiting 16:39:32 (4592): No heartbeat from core client for 30 sec - exiting 16:39:33 (4592): No heartbeat from core client for 30 sec - exiting 16:39:34 (4592): No heartbeat from core client for 30 sec - exiting 16:39:35 (4592): No heartbeat from core client for 30 sec - exiting 16:39:36 (4592): No heartbeat from core client for 30 sec - exiting 16:39:37 (4592): No heartbeat from core client for 30 sec - exiting 16:39:38 (4592): No heartbeat from core client for 30 sec - exiting 16:39:39 (4592): No heartbeat from core client for 30 sec - exiting 16:39:40 (4592): No heartbeat from core client for 30 sec - exiting 16:39:41 (4592): No heartbeat from core client for 30 sec - exiting 16:39:42 (4592): No heartbeat from core client for 30 sec - exiting 16:39:43 (4592): No heartbeat from core client for 30 sec - exiting 16:39:44 (4592): No heartbeat from core client for 30 sec - exiting 16:39:45 (4592): No heartbeat from core client for 30 sec - exiting 16:39:46 (4592): No heartbeat from core client for 30 sec - exiting 16:39:47 (4592): No heartbeat from core client for 30 sec - exiting 16:39:48 (4592): No heartbeat from core client for 30 sec - exiting 16:39:49 (4592): No heartbeat from core client for 30 sec - exiting 16:39:50 (4592): No heartbeat from core client for 30 sec - exiting 16:39:51 (4592): No heartbeat from core client for 30 sec - exiting 16:39:52 (4592): No heartbeat from core client for 30 sec - exiting 16:39:53 (4592): No heartbeat from core client for 30 sec - exiting 16:39:54 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:55 (4592): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 16:58:15 (5828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:06 (4896): No heartbeat from core client for 30 sec - exiting 09:40:07 (4896): No heartbeat from core client for 30 sec - exiting 09:40:08 (4896): No heartbeat from core client for 30 sec - exiting 09:40:09 (4896): No heartbeat from core client for 30 sec - exiting 09:40:10 (4896): No heartbeat from core client for 30 sec - exiting 09:40:11 (4896): No heartbeat from core client for 30 sec - exiting 09:40:12 (4896): No heartbeat from core client for 30 sec - exiting 09:40:13 (4896): No heartbeat from core client for 30 sec - exiting 09:40:14 (4896): No heartbeat from core client for 30 sec - exiting 09:40:15 (4896): No heartbeat from core client for 30 sec - exiting 09:40:16 (4896): No heartbeat from core client for 30 sec - exiting 09:40:17 (4896): No heartbeat from core client for 30 sec - exiting 09:40:18 (4896): No heartbeat from core client for 30 sec - exiting 09:40:19 (4896): No heartbeat from core client for 30 sec - exiting 09:40:20 (4896): No heartbeat from core client for 30 sec - exiting 09:40:21 (4896): No heartbeat from core client for 30 sec - exiting 09:40:22 (4896): No heartbeat from core client for 30 sec - exiting 09:40:23 (4896): No heartbeat from core client for 30 sec - exiting 09:40:24 (4896): No heartbeat from core client for 30 sec - exiting 09:40:25 (4896): No heartbeat from core client for 30 sec - exiting 09:40:26 (4896): No heartbeat from core client for 30 sec - exiting 09:40:27 (4896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:20:26 (4028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:01:19 (6424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:03:30 (5792): No heartbeat from core client for 30 sec - exiting 10:03:31 (5792): No heartbeat from core client for 30 sec - exiting 10:03:32 (5792): No heartbeat from core client for 30 sec - exiting 10:03:33 (5792): No heartbeat from core client for 30 sec - exiting 10:03:34 (5792): No heartbeat from core client for 30 sec - exiting 10:03:35 (5792): No heartbeat from core client for 30 sec - exiting 10:03:36 (5792): No heartbeat from core client for 30 sec - exiting 10:03:37 (5792): No heartbeat from core client for 30 sec - exiting 10:03:38 (5792): No heartbeat from core client for 30 sec - exiting 10:03:39 (5792): No heartbeat from core client for 30 sec - exiting 10:03:40 (5792): No heartbeat from core client for 30 sec - exiting 10:03:41 (5792): No heartbeat from core client for 30 sec - exiting 10:03:42 (5792): No heartbeat from core client for 30 sec - exiting 10:03:43 (5792): No heartbeat from core client for 30 sec - exiting 10:03:44 (5792): No heartbeat from core client for 30 sec - exiting 10:03:45 (5792): No heartbeat from core client for 30 sec - exiting 10:03:46 (5792): No heartbeat from core client for 30 sec - exiting 10:03:47 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:48 (5792): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B77373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 19:27:06 (5896): No heartbeat from core client for 30 sec - exiting 19:27:07 (5896): No heartbeat from core client for 30 sec - exiting 19:27:08 (5896): No heartbeat from core client for 30 sec - exiting 19:27:09 (5896): No heartbeat from core client for 30 sec - exiting 19:27:10 (5896): No heartbeat from core client for 30 sec - exiting 19:27:11 (5896): No heartbeat from core client for 30 sec - exiting 19:27:12 (5896): No heartbeat from core client for 30 sec - exiting 19:27:13 (5896): No heartbeat from core client for 30 sec - exiting 19:27:14 (5896): No heartbeat from core client for 30 sec - exiting 19:27:15 (5896): No heartbeat from core client for 30 sec - exiting 19:27:16 (5896): No heartbeat from core client for 30 sec - exiting 19:27:17 (5896): No heartbeat from core client for 30 sec - exiting 19:27:18 (5896): No heartbeat from core client for 30 sec - exiting 19:27:19 (5896): No heartbeat from core client for 30 sec - exiting 19:27:20 (5896): No heartbeat from core client for 30 sec - exiting 19:27:21 (5896): No heartbeat from core client for 30 sec - exiting 19:27:22 (5896): No heartbeat from core client for 30 sec - exiting 19:27:23 (5896): No heartbeat from core client for 30 sec - exiting 19:27:24 (5896): No heartbeat from core client for 30 sec - exiting 19:27:25 (5896): No heartbeat from core client for 30 sec - exiting 19:27:26 (5896): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778B3AB3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_n0eu_1920_40_008362365/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Jun 2013 09:39:36	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	518,400	505,713	0.9755
31 May 2013 09:31:34	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	492,480	479,262	0.9732
30 May 2013 09:44:31	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	466,560	454,469	0.9741
29 May 2013 10:53:38	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	440,640	429,626	0.9750
28 May 2013 09:08:50	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	414,720	403,870	0.9738
28 May 2013 02:06:12	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	388,800	378,866	0.9744
27 May 2013 20:07:27	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	362,880	354,577	0.9771
27 May 2013 09:44:11	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	336,960	328,909	0.9761
24 May 2013 17:36:08	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	311,040	302,776	0.9734
24 May 2013 10:08:20	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	285,120	276,176	0.9686
22 May 2013 18:06:48	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	259,200	250,137	0.9650
21 May 2013 16:31:53	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	233,280	223,552	0.9583
20 May 2013 15:18:12	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	207,360	197,728	0.9535
18 May 2013 11:44:45	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	181,440	171,558	0.9455
17 May 2013 11:56:33	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	155,520	145,857	0.9379
16 May 2013 16:26:37	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	129,600	120,560	0.9302
15 May 2013 11:06:21	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	103,680	95,183	0.9180
14 May 2013 15:21:04	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	77,760	70,225	0.9031
13 May 2013 16:03:09	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	51,840	45,273	0.8733
11 May 2013 12:09:18	1245161	15771139	hadcm3n_n0eu_1920_40_008362365_0	25,920	23,598	0.9104