Task 13559811

Name	hadcm3n_ye9g_1900_40_007527019_0
Workunit	7724494
Created	28 Oct 2011, 13:48:51 UTC
Sent	29 Oct 2011, 3:28:21 UTC
Report deadline	28 Jan 2012, 10:55:32 UTC
Received	16 Nov 2011, 5:55:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1161345
Run time	4 days 12 hours 55 min 58 sec
CPU time	4 days 9 hours 55 min 39 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4476, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:23:34 (4408): No heartbeat from core client for 30 sec - exiting 19:23:35 (4408): No heartbeat from core client for 30 sec - exiting 19:23:36 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:56 (4956): No heartbeat from core client for 30 sec - exiting 08:48:57 (4956): No heartbeat from core client for 30 sec - exiting 08:48:58 (4956): No heartbeat from core client for 30 sec - exiting 08:48:59 (4956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:18 (1620): No heartbeat from core client for 30 sec - exiting 18:24:20 (1620): No heartbeat from core client for 30 sec - exiting 18:24:21 (1620): No heartbeat from core client for 30 sec - exiting 18:24:22 (1620): No heartbeat from core client for 30 sec - exiting 18:24:23 (1620): No heartbeat from core client for 30 sec - exiting 18:24:24 (1620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2748, iMonCtr=1 Model crash detected, will try to restart... 10:33:09 (4892): No heartbeat from core client for 30 sec - exiting 10:33:10 (4892): No heartbeat from core client for 30 sec - exiting 10:33:12 (4892): No heartbeat from core client for 30 sec - exiting 10:33:13 (4892): No heartbeat from core client for 30 sec - exiting 10:33:14 (4892): No heartbeat from core client for 30 sec - exiting 10:33:15 (4892): No heartbeat from core client for 30 sec - exiting 10:33:16 (4892): No heartbeat from core client for 30 sec - exiting 10:33:17 (4892): No heartbeat from core client for 30 sec - exiting 10:33:18 (4892): No heartbeat from core client for 30 sec - exiting 10:33:19 (4892): No heartbeat from core client for 30 sec - exiting 10:33:20 (4892): No heartbeat from core client for 30 sec - exiting 10:33:21 (4892): No heartbeat from core client for 30 sec - exiting 10:33:22 (4892): No heartbeat from core client for 30 sec - exiting 10:33:23 (4892): No heartbeat from core client for 30 sec - exiting 10:33:25 (4892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:04:48 (3272): No heartbeat from core client for 30 sec - exiting 09:04:49 (3272): No heartbeat from core client for 30 sec - exiting 09:04:50 (3272): No heartbeat from core client for 30 sec - exiting 09:04:51 (3272): No heartbeat from core client for 30 sec - exiting 09:04:52 (3272): No heartbeat from core client for 30 sec - exiting 09:04:53 (3272): No heartbeat from core client for 30 sec - exiting 09:04:54 (3272): No heartbeat from core client for 30 sec - exiting 09:04:55 (3272): No heartbeat from core client for 30 sec - exiting 09:04:57 (3272): No heartbeat from core client for 30 sec - exiting 09:04:58 (3272): No heartbeat from core client for 30 sec - exiting 09:04:59 (3272): No heartbeat from core client for 30 sec - exiting 09:05:00 (3272): No heartbeat from core client for 30 sec - exiting 09:05:01 (3272): No heartbeat from core client for 30 sec - exiting 09:05:02 (3272): No heartbeat from core client for 30 sec - exiting 09:05:03 (3272): No heartbeat from core client for 30 sec - exiting 09:05:04 (3272): No heartbeat from core client for 30 sec - exiting 09:05:05 (3272): No heartbeat from core client for 30 sec - exiting 09:05:06 (3272): No heartbeat from core client for 30 sec - exiting 09:05:07 (3272): No heartbeat from core client for 30 sec - exiting 09:05:09 (3272): No heartbeat from core client for 30 sec - exiting 09:05:10 (3272): No heartbeat from core client for 30 sec - exiting 09:05:11 (3272): No heartbeat from core client for 30 sec - exiting 09:05:12 (3272): No heartbeat from core client for 30 sec - exiting 09:05:13 (3272): No heartbeat from core client for 30 sec - exiting 09:05:14 (3272): No heartbeat from core client for 30 sec - exiting 09:05:15 (3272): No heartbeat from core client for 30 sec - exiting 09:05:16 (3272): No heartbeat from core client for 30 sec - exiting 09:05:17 (3272): No heartbeat from core client for 30 sec - exiting 09:05:18 (3272): No heartbeat from core client for 30 sec - exiting 09:05:19 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:28 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:24:13 (4388): No heartbeat from core client for 30 sec - exiting 10:24:14 (4388): No heartbeat from core client for 30 sec - exiting 10:24:15 (4388): No heartbeat from core client for 30 sec - exiting 10:24:16 (4388): No heartbeat from core client for 30 sec - exiting 10:24:17 (4388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3040, iMonCtr=1 Model crash detected, will try to restart... 10:03:14 (3144): No heartbeat from core client for 30 sec - exiting 10:03:16 (3144): No heartbeat from core client for 30 sec - exiting 10:03:17 (3144): No heartbeat from core client for 30 sec - exiting 10:03:18 (3144): No heartbeat from core client for 30 sec - exiting 10:03:19 (3144): No heartbeat from core client for 30 sec - exiting 10:03:20 (3144): No heartbeat from core client for 30 sec - exiting 10:03:21 (3144): No heartbeat from core client for 30 sec - exiting 10:03:22 (3144): No heartbeat from core client for 30 sec - exiting 10:03:23 (3144): No heartbeat from core client for 30 sec - exiting 10:03:24 (3144): No heartbeat from core client for 30 sec - exiting 10:03:25 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:27 (3144): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77AFB84B write attempt to address 0x40EC0142 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x00417B59 read attempt to address 0x0C0000D6 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ye9g_1900_40_007527019/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Nov 2011 05:58:29	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	259,200	375,898	1.4502
16 Nov 2011 05:58:29	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	233,280	338,403	1.4506
16 Nov 2011 05:58:29	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	207,360	301,202	1.4526
16 Nov 2011 05:58:29	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	181,440	263,916	1.4546
16 Nov 2011 05:58:29	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	155,520	226,619	1.4572
08 Nov 2011 11:11:17	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	129,600	188,964	1.4581
08 Nov 2011 00:45:00	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	103,680	151,398	1.4602
06 Nov 2011 12:19:04	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	77,760	113,888	1.4646
06 Nov 2011 00:27:27	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	51,840	76,541	1.4765
04 Nov 2011 11:42:57	1161345	13559811	hadcm3n_ye9g_1900_40_007527019_0	25,920	38,438	1.4829