Task 15801552

Name	hadcm3n_o7mn_2020_40_008373160_0
Workunit	8524019
Created	29 May 2013, 18:30:03 UTC
Sent	29 May 2013, 18:33:50 UTC
Report deadline	29 Aug 2013, 2:01:01 UTC
Received	13 Jun 2013, 15:45:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1246737
Run time	8 days 4 hours 38 min 29 sec
CPU time	7 days 1 hours 12 min 45 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 11:47:35 (508): No heartbeat from core client for 30 sec - exiting 11:47:37 (508): No heartbeat from core client for 30 sec - exiting 11:47:38 (508): No heartbeat from core client for 30 sec - exiting 11:47:39 (508): No heartbeat from core client for 30 sec - exiting 11:47:40 (508): No heartbeat from core client for 30 sec - exiting 11:47:41 (508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:47:42 (508): No heartbeat from core client for 30 sec - exiting 11:47:43 (508): No heartbeat from core client for 30 sec - exiting 11:47:44 (508): No heartbeat from core client for 30 sec - exiting 11:47:45 (508): No heartbeat from core client for 30 sec - exiting 11:48:30 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:49:21 (2804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:48:35 (2568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:21:51 (1140): No heartbeat from core client for 30 sec - exiting 12:21:53 (1140): No heartbeat from core client for 30 sec - exiting 12:21:54 (1140): No heartbeat from core client for 30 sec - exiting 12:21:56 (1140): No heartbeat from core client for 30 sec - exiting 12:21:57 (1140): No heartbeat from core client for 30 sec - exiting 12:21:58 (1140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:59 (1140): No heartbeat from core client for 30 sec - exiting 12:22:00 (1140): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 21:44:43 (2252): No heartbeat from core client for 30 sec - exiting 21:44:45 (2252): No heartbeat from core client for 30 sec - exiting 21:44:46 (2252): No heartbeat from core client for 30 sec - exiting 21:44:47 (2252): No heartbeat from core client for 30 sec - exiting 21:44:48 (2252): No heartbeat from core client for 30 sec - exiting 21:44:49 (2252): No heartbeat from core client for 30 sec - exiting 21:44:50 (2252): No heartbeat from core client for 30 sec - exiting 21:44:51 (2252): No heartbeat from core client for 30 sec - exiting 21:44:52 (2252): No heartbeat from core client for 30 sec - exiting 21:44:53 (2252): No heartbeat from core client for 30 sec - exiting 21:44:54 (2252): No heartbeat from core client for 30 sec - exiting 21:44:56 (2252): No heartbeat from core client for 30 sec - exiting 21:44:57 (2252): No heartbeat from core client for 30 sec - exiting 21:44:58 (2252): No heartbeat from core client for 30 sec - exiting 21:44:59 (2252): No heartbeat from core client for 30 sec - exiting 21:45:00 (2252): No heartbeat from core client for 30 sec - exiting 21:45:01 (2252): No heartbeat from core client for 30 sec - exiting 21:45:02 (2252): No heartbeat from core client for 30 sec - exiting 21:45:03 (2252): No heartbeat from core client for 30 sec - exiting 21:45:04 (2252): No heartbeat from core client for 30 sec - exiting 21:45:05 (2252): No heartbeat from core client for 30 sec - exiting 21:45:06 (2252): No heartbeat from core client for 30 sec - exiting 21:45:08 (2252): No heartbeat from core client for 30 sec - exiting 21:45:09 (2252): No heartbeat from core client for 30 sec - exiting 21:45:10 (2252): No heartbeat from core client for 30 sec - exiting 21:45:11 (2252): No heartbeat from core client for 30 sec - exiting 21:45:12 (2252): No heartbeat from core client for 30 sec - exiting 21:45:13 (2252): No heartbeat from core client for 30 sec - exiting 21:45:14 (2252): No heartbeat from core client for 30 sec - exiting 21:45:15 (2252): No heartbeat from core client for 30 sec - exiting 21:45:16 (2252): No heartbeat from core client for 30 sec - exiting 21:45:17 (2252): No heartbeat from core client for 30 sec - exiting 21:45:18 (2252): No heartbeat from core client for 30 sec - exiting 21:45:20 (2252): No heartbeat from core client for 30 sec - exiting 21:45:21 (2252): No heartbeat from core client for 30 sec - exiting 21:45:22 (2252): No heartbeat from core client for 30 sec - exiting 21:45:23 (2252): No heartbeat from core client for 30 sec - exiting 21:45:24 (2252): No heartbeat from core client for 30 sec - exiting 21:45:25 (2252): No heartbeat from core client for 30 sec - exiting 21:45:26 (2252): No heartbeat from core client for 30 sec - exiting 21:45:27 (2252): No heartbeat from core client for 30 sec - exiting 21:45:28 (2252): No heartbeat from core client for 30 sec - exiting 21:45:29 (2252): No heartbeat from core client for 30 sec - exiting 21:45:30 (2252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:58:03 (2612): No heartbeat from core client for 30 sec - exiting 22:58:04 (2612): No heartbeat from core client for 30 sec - exiting 22:58:05 (2612): No heartbeat from core client for 30 sec - exiting 22:58:06 (2612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:58:08 (2612): No heartbeat from core client for 30 sec - exiting 22:58:09 (2612): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2044, iMonCtr=1 Model crash detected, will try to restart... 09:07:21 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:00 (3440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:41:32 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:06 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:24:13 (1456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:12 (1332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... C09:04:03 (3316): No heartbeat from core client for 30 sec - exiting 09:04:04 (3316): No heartbeat from core client for 30 sec - exiting 09:04:05 (3316): No heartbeat from core client for 30 sec - exiting 09:04:06 (3316): No heartbeat from core client for 30 sec - exiting 09:04:07 (3316): No heartbeat from core client for 30 sec - exiting 09:04:09 (3316): No heartbeat from core client for 30 sec - exiting 09:04:10 (3316): No heartbeat from core client for 30 sec - exiting 09:04:11 (3316): No heartbeat from core client for 30 sec - exiting 09:04:12 (3316): No heartbeat from core client for 30 sec - exiting 09:04:13 (3316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:14 (3316): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:24:22 (4036): No heartbeat from core client for 30 sec - exiting 10:24:24 (4036): No heartbeat from core client for 30 sec - exiting 10:24:25 (4036): No heartbeat from core client for 30 sec - exiting 10:24:26 (4036): No heartbeat from core client for 30 sec - exiting 10:24:27 (4036): No heartbeat from core client for 30 sec - exiting 10:24:28 (4036): No heartbeat from core client for 30 sec - exiting 10:24:29 (4036): No heartbeat from core client for 30 sec - exiting 10:24:30 (4036): No heartbeat from core client for 30 sec - exiting 10:24:31 (4036): No heartbeat from core client for 30 sec - exiting 10:24:32 (4036): No heartbeat from core client for 30 sec - exiting 10:24:33 (4036): No heartbeat from core client for 30 sec - exiting 10:24:35 (4036): No heartbeat from core client for 30 sec - exiting 10:24:36 (4036): No heartbeat from core client for 30 sec - exiting 10:24:37 (4036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:49 (3040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:41 (2484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3976, iMonCtr=1 Model crash detected, will try to restart... 10:59:00 (2340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:15 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1972, iMonCtr=1 Model crash detected, will try to restart... 08:51:03 (1864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:51:48 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:13:24 (2296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:20 (2964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2932, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77827223 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file D:\BOINCFiles/projects/climateprediction.net/hadcm3n_o7mn_2020_40_008373160/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jun 2013 14:31:27	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	259,200	609,157	2.3501
11 Jun 2013 21:29:45	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	233,280	552,618	2.3689
10 Jun 2013 14:25:04	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	207,360	493,707	2.3809
09 Jun 2013 09:19:54	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	181,440	432,736	2.3850
07 Jun 2013 21:06:58	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	155,520	370,863	2.3847
06 Jun 2013 11:12:25	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	129,600	310,902	2.3989
05 Jun 2013 00:36:52	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	103,680	251,243	2.4233
03 Jun 2013 18:54:25	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	77,760	191,216	2.4591
01 Jun 2013 17:24:09	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	51,840	122,926	2.3713
31 May 2013 01:33:26	1246737	15801552	hadcm3n_o7mn_2020_40_008373160_0	25,920	59,284	2.2872