Task 15716371

Name	hadcm3n_3gca_1980_40_008340117_2
Workunit	8490978
Created	8 Apr 2013, 2:48:16 UTC
Sent	8 Apr 2013, 2:52:55 UTC
Report deadline	8 Jul 2013, 10:20:06 UTC
Received	28 Apr 2013, 18:09:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1154356
Run time	10 days 6 hours 50 min 36 sec
CPU time	9 days 17 hours 51 min 43 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.23 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 19:48:42 (3468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:38 (1844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:32 (1388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:24 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:18 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:14 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:15 (892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:09:08 (1228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:57:41 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:56:35 (780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:53:53 (3732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6008, iMonCtr=1 Model crash detected, will try to restart... 19:37:43 (2768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:36:26 (1096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:35:41 (3160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:34:59 (3820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:33:46 (724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:32:40 (1092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:48:56 (3372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:45:57 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:41:45 (5036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:46 (6588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:26:05 (6588): No heartbeat from core client for 30 sec - exiting 23:26:14 (6588): No heartbeat from core client for 30 sec - exiting 23:26:20 (6588): No heartbeat from core client for 30 sec - exiting 23:26:22 (6588): No heartbeat from core client for 30 sec - exiting 23:26:23 (6588): No heartbeat from core client for 30 sec - exiting 23:26:24 (6588): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3652, iMonCtr=1 Model crash detected, will try to restart... 19:39:14 (3120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:22 (3864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:19 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:13 (4228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:32 (3636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:49 (3900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:31 (3572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:30 (3856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:55 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:39:36 (1092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:32:51 (2116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:41 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:04 (3188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:52:50 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:15 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:29 (3928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:24:31 (4092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:22:28 (504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:43:03 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:03 (3516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:33 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:23 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:06:35 (3160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:16 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:00 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:05 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:53 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:18:49 (8724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:40 (8312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:00:00 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77A63F99 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3gca_1980_40_008340117/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Apr 2013 18:17:21	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	518,400	841,898	1.6240
27 Apr 2013 22:22:34	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	492,480	798,350	1.6211
27 Apr 2013 09:03:03	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	466,560	754,002	1.6161
26 Apr 2013 03:11:08	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	440,640	710,928	1.6134
25 Apr 2013 03:20:10	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	414,720	667,473	1.6095
24 Apr 2013 02:43:08	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	388,800	624,126	1.6053
23 Apr 2013 03:02:10	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	362,880	580,769	1.6004
22 Apr 2013 03:47:05	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	336,960	536,285	1.5915
19 Apr 2013 05:47:41	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	311,040	490,000	1.5754
18 Apr 2013 17:02:11	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	285,120	446,010	1.5643
17 Apr 2013 17:28:41	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	259,200	405,161	1.5631
16 Apr 2013 19:25:23	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	233,280	367,423	1.5750
15 Apr 2013 20:36:11	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	207,360	329,092	1.5871
14 Apr 2013 23:02:14	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	181,440	290,891	1.6032
13 Apr 2013 23:08:03	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	155,520	251,789	1.6190
13 Apr 2013 08:58:30	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	129,600	213,065	1.6440
12 Apr 2013 04:07:46	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	103,680	172,147	1.6604
11 Apr 2013 03:42:28	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	77,760	129,382	1.6639
10 Apr 2013 03:26:11	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	51,840	86,733	1.6731
09 Apr 2013 03:18:03	1154356	15716371	hadcm3n_3gca_1980_40_008340117_2	25,920	43,929	1.6948