Task 15819490

Name	hadcm3n_4f12_1940_40_008308418_3
Workunit	8459553
Created	3 Jun 2013, 0:02:01 UTC
Sent	12 Jun 2013, 20:24:13 UTC
Report deadline	12 Sep 2013, 3:51:24 UTC
Received	8 Jul 2013, 20:41:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1186368
Run time	20 days 6 hours 42 min 46 sec
CPU time	18 days 2 hours 49 min 59 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1196, iMonCtr=1 Model crash detected, will try to restart... 02:56:34 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:52 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:44:43 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:49:38 (7120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:56:36 (6908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:22:36 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:23:50 (2088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:25:42 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:40 (2352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:24 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:46:53 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:01:22 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5956, iMonCtr=1 Model crash detected, will try to restart... 13:31:45 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x777E3AB3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 23:18:31 (2104): No heartbeat from core client for 30 sec - exiting 23:18:32 (2104): No heartbeat from core client for 30 sec - exiting 23:18:33 (2104): No heartbeat from core client for 30 sec - exiting 23:18:34 (2104): No heartbeat from core client for 30 sec - exiting 23:18:35 (2104): No heartbeat from core client for 30 sec - exiting 23:18:36 (2104): No heartbeat from core client for 30 sec - exiting 23:18:37 (2104): No heartbeat from core client for 30 sec - exiting 23:18:38 (2104): No heartbeat from core client for 30 sec - exiting 23:18:39 (2104): No heartbeat from core client for 30 sec - exiting 23:18:40 (2104): No heartbeat from core client for 30 sec - exiting 23:18:41 (2104): No heartbeat from core client for 30 sec - exiting 23:18:42 (2104): No heartbeat from core client for 30 sec - exiting 23:18:43 (2104): No heartbeat from core client for 30 sec - exiting 23:18:44 (2104): No heartbeat from core client for 30 sec - exiting 23:18:45 (2104): No heartbeat from core client for 30 sec - exiting 23:18:46 (2104): No heartbeat from core client for 30 sec - exiting 23:18:47 (2104): No heartbeat from core client for 30 sec - exiting 23:18:48 (2104): No heartbeat from core client for 30 sec - exiting 23:18:49 (2104): No heartbeat from core client for 30 sec - exiting 23:18:50 (2104): No heartbeat from core client for 30 sec - exiting 23:18:51 (2104): No heartbeat from core client for 30 sec - exiting 23:18:52 (2104): No heartbeat from core client for 30 sec - exiting 23:18:53 (2104): No heartbeat from core client for 30 sec - exiting 23:18:54 (2104): No heartbeat from core client for 30 sec - exiting 23:18:55 (2104): No heartbeat from core client for 30 sec - exiting 23:18:56 (2104): No heartbeat from core client for 30 sec - exiting 23:18:57 (2104): No heartbeat from core client for 30 sec - exiting 23:18:58 (2104): No heartbeat from core client for 30 sec - exiting 23:18:59 (2104): No heartbeat from core client for 30 sec - exiting 23:19:00 (2104): No heartbeat from core client for 30 sec - exiting 23:19:01 (2104): No heartbeat from core client for 30 sec - exiting 23:19:02 (2104): No heartbeat from core client for 30 sec - exiting 23:19:03 (2104): No heartbeat from core client for 30 sec - exiting 23:19:04 (2104): No heartbeat from core client for 30 sec - exiting 23:19:05 (2104): No heartbeat from core client for 30 sec - exiting 23:19:06 (2104): No heartbeat from core client for 30 sec - exiting 23:19:07 (2104): No heartbeat from core client for 30 sec - exiting 23:19:08 (2104): No heartbeat from core client for 30 sec - exiting 23:19:09 (2104): No heartbeat from core client for 30 sec - exiting 23:19:10 (2104): No heartbeat from core client for 30 sec - exiting 23:19:11 (2104): No heartbeat from core client for 30 sec - exiting 23:19:12 (2104): No heartbeat from core client for 30 sec - exiting 23:19:13 (2104): No heartbeat from core client for 30 sec - exiting 23:19:14 (2104): No heartbeat from core client for 30 sec - exiting 23:19:15 (2104): No heartbeat from core client for 30 sec - exiting 23:19:16 (2104): No heartbeat from core client for 30 sec - exiting 23:19:17 (2104): No heartbeat from core client for 30 sec - exiting 23:19:18 (2104): No heartbeat from core client for 30 sec - exiting 23:19:19 (2104): No heartbeat from core client for 30 sec - exiting 23:19:20 (2104): No heartbeat from core client for 30 sec - exiting 23:19:21 (2104): No heartbeat from core client for 30 sec - exiting 23:19:22 (2104): No heartbeat from core client for 30 sec - exiting 23:19:23 (2104): No heartbeat from core client for 30 sec - exiting 23:19:24 (2104): No heartbeat from core client for 30 sec - exiting 23:19:25 (2104): No heartbeat from core client for 30 sec - exiting 23:19:26 (2104): No heartbeat from core client for 30 sec - exiting 23:19:27 (2104): No heartbeat from core client for 30 sec - exiting 23:19:28 (2104): No heartbeat from core client for 30 sec - exiting 23:19:29 (2104): No heartbeat from core client for 30 sec - exiting 23:19:30 (2104): No heartbeat from core client for 30 sec - exiting 23:19:31 (2104): No heartbeat from core client for 30 sec - exiting 23:19:32 (2104): No heartbeat from core client for 30 sec - exiting 23:19:33 (2104): No heartbeat from core client for 30 sec - exiting 23:19:34 (2104): No heartbeat from core client for 30 sec - exiting 23:19:35 (2104): No heartbeat from core client for 30 sec - exiting 23:19:36 (2104): No heartbeat from core client for 30 sec - exiting 23:19:37 (2104): No heartbeat from core client for 30 sec - exiting 23:19:38 (2104): No heartbeat from core client for 30 sec - exiting 23:19:39 (2104): No heartbeat from core client for 30 sec - exiting 23:19:40 (2104): No heartbeat from core client for 30 sec - exiting 23:19:41 (2104): No heartbeat from core client for 30 sec - exiting 23:19:42 (2104): No heartbeat from core client for 30 sec - exiting 23:19:43 (2104): No heartbeat from core client for 30 sec - exiting 23:19:44 (2104): No heartbeat from core client for 30 sec - exiting 23:19:45 (2104): No heartbeat from core client for 30 sec - exiting 23:19:46 (2104): No heartbeat from core client for 30 sec - exiting 23:19:47 (2104): No heartbeat from core client for 30 sec - exiting 23:19:48 (2104): No heartbeat from core client for 30 sec - exiting 23:19:49 (2104): No heartbeat from core client for 30 sec - exiting 23:19:50 (2104): No heartbeat from core client for 30 sec - exiting 23:19:51 (2104): No heartbeat from core client for 30 sec - exiting 23:19:52 (2104): No heartbeat from core client for 30 sec - exiting 23:19:53 (2104): No heartbeat from core client for 30 sec - exiting 23:19:54 (2104): No heartbeat from core client for 30 sec - exiting 23:19:55 (2104): No heartbeat from core client for 30 sec - exiting 23:19:56 (2104): No heartbeat from core client for 30 sec - exiting 23:19:57 (2104): No heartbeat from core client for 30 sec - exiting 23:19:58 (2104): No heartbeat from core client for 30 sec - exiting 23:19:59 (2104): No heartbeat from core client for 30 sec - exiting 23:20:00 (2104): No heartbeat from core client for 30 sec - exiting 23:20:01 (2104): No heartbeat from core client for 30 sec - exiting 23:20:02 (2104): No heartbeat from core client for 30 sec - exiting 23:20:03 (2104): No heartbeat from core client for 30 sec - exiting 23:20:04 (2104): No heartbeat from core client for 30 sec - exiting 23:20:05 (2104): No heartbeat from core client for 30 sec - exiting 23:20:06 (2104): No heartbeat from core client for 30 sec - exiting 23:20:07 (2104): No heartbeat from core client for 30 sec - exiting 23:20:08 (2104): No heartbeat from core client for 30 sec - exiting 23:20:09 (2104): No heartbeat from core client for 30 sec - exiting 23:20:10 (2104): No heartbeat from core client for 30 sec - exiting 23:20:11 (2104): No heartbeat from core client for 30 sec - exiting 23:20:12 (2104): No heartbeat from core client for 30 sec - exiting 23:20:13 (2104): No heartbeat from core client for 30 sec - exiting 23:20:14 (2104): No heartbeat from core client for 30 sec - exiting 23:20:15 (2104): No heartbeat from core client for 30 sec - exiting 23:20:16 (2104): No heartbeat from core client for 30 sec - exiting 23:20:17 (2104): No heartbeat from core client for 30 sec - exiting 23:20:18 (2104): No heartbeat from core client for 30 sec - exiting 23:20:19 (2104): No heartbeat from core client for 30 sec - exiting 23:20:20 (2104): No heartbeat from core client for 30 sec - exiting 23:20:21 (2104): No heartbeat from core client for 30 sec - exiting 23:20:22 (2104): No heartbeat from core client for 30 sec - exiting 23:20:23 (2104): No heartbeat from core client for 30 sec - exiting 23:20:24 (2104): No heartbeat from core client for 30 sec - exiting 23:20:25 (2104): No heartbeat from core client for 30 sec - exiting 23:20:26 (2104): No heartbeat from core client for 30 sec - exiting 23:20:27 (2104): No heartbeat from core client for 30 sec - exiting 23:20:28 (2104): No heartbeat from core client for 30 sec - exiting 23:20:29 (2104): No heartbeat from core client for 30 sec - exiting 23:20:30 (2104): No heartbeat from core client for 30 sec - exiting 23:20:31 (2104): No heartbeat from core client for 30 sec - exiting 23:20:32 (2104): No heartbeat from core client for 30 sec - exiting 23:20:33 (2104): No heartbeat from core client for 30 sec - exiting 23:20:34 (2104): No heartbeat from core client for 30 sec - exiting 23:20:35 (2104): No heartbeat from core client for 30 sec - exiting 23:20:36 (2104): No heartbeat from core client for 30 sec - exiting 23:20:37 (2104): No heartbeat from core client for 30 sec - exiting 23:20:38 (2104): No heartbeat from core client for 30 sec - exiting 23:20:39 (2104): No heartbeat from core client for 30 sec - exiting 23:20:40 (2104): No heartbeat from core client for 30 sec - exiting 23:20:41 (2104): No heartbeat from core client for 30 sec - exiting 23:20:42 (2104): No heartbeat from core client for 30 sec - exiting 23:20:43 (2104): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x777E3AB3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_4f12_1940_40_008308418/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2013 19:32:49	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	518,400	1,538,569	2.9679
06 Jul 2013 17:09:58	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	492,480	1,461,968	2.9686
06 Jul 2013 05:18:09	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	466,560	1,386,297	2.9713
03 Jul 2013 21:00:05	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	440,640	1,310,040	2.9730
02 Jul 2013 19:55:51	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	414,720	1,231,726	2.9700
02 Jul 2013 11:52:07	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	388,800	1,151,696	2.9622
02 Jul 2013 10:52:03	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	362,880	1,072,305	2.9550
02 Jul 2013 10:12:46	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	336,960	991,816	2.9434
28 Jun 2013 16:40:43	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	311,040	911,745	2.9313
27 Jun 2013 15:21:44	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	285,120	834,321	2.9262
26 Jun 2013 16:23:37	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	259,200	759,375	2.9297
25 Jun 2013 16:10:52	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	233,280	683,145	2.9284
24 Jun 2013 10:15:40	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	207,360	606,296	2.9239
23 Jun 2013 03:24:27	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	181,440	530,795	2.9255
21 Jun 2013 15:51:37	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	155,520	455,199	2.9269
19 Jun 2013 13:43:33	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	129,600	378,722	2.9222
18 Jun 2013 09:39:26	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	103,680	303,020	2.9226
16 Jun 2013 10:36:35	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	77,760	228,506	2.9386
14 Jun 2013 22:39:34	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	51,840	152,430	2.9404
13 Jun 2013 22:46:53	1186368	15819490	hadcm3n_4f12_1940_40_008308418_3	25,920	75,940	2.9298