Task 15799022

Name	hadcm3n_n4on_1920_40_008321636_2
Workunit	8472771
Created	27 May 2013, 18:03:42 UTC
Sent	27 May 2013, 18:04:00 UTC
Report deadline	27 Aug 2013, 1:31:11 UTC
Received	28 Oct 2013, 1:41:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1281494
Run time	15 days 22 hours 7 min 3 sec
CPU time	15 days 8 hours 53 min 32 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:10:34 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:35 (4216): No heartbeat from core client for 30 sec - exiting 21:10:36 (4216): No heartbeat from core client for 30 sec - exiting 21:10:37 (4216): No heartbeat from core client for 30 sec - exiting 21:10:38 (4216): No heartbeat from core client for 30 sec - exiting 21:10:39 (4216): No heartbeat from core client for 30 sec - exiting 21:10:40 (4216): No heartbeat from core client for 30 sec - exiting 21:10:41 (4216): No heartbeat from core client for 30 sec - exiting 21:10:42 (4216): No heartbeat from core client for 30 sec - exiting 21:10:43 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5412, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=1 Model crash detected, will try to restart... 11:35:38 (1376): No heartbeat from core client for 30 sec - exiting 11:35:39 (1376): No heartbeat from core client for 30 sec - exiting 11:35:40 (1376): No heartbeat from core client for 30 sec - exiting 11:35:41 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5800, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5736, iMonCtr=1 Model crash detected, will try to restart... 15:45:21 (1804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5708, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5652, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 17:23:36 (5284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5184, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... 17:24:16 (4940): No heartbeat from core client for 30 sec - exiting 17:24:17 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5360, iMonCtr=1 Model crash detected, will try to restart... 13:49:42 (6084): No heartbeat from core client for 30 sec - exiting 13:49:43 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5852, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5652, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... 15:20:29 (5316): No heartbeat from core client for 30 sec - exiting 15:20:30 (5316): No heartbeat from core client for 30 sec - exiting 15:20:31 (5316): No heartbeat from core client for 30 sec - exiting 15:20:32 (5316): No heartbeat from core client for 30 sec - exiting 15:20:33 (5316): No heartbeat from core client for 30 sec - exiting 15:20:34 (5316): No heartbeat from core client for 30 sec - exiting 15:20:36 (5316): No heartbeat from core client for 30 sec - exiting 15:20:37 (5316): No heartbeat from core client for 30 sec - exiting 15:20:38 (5316): No heartbeat from core client for 30 sec - exiting 15:20:39 (5316): No heartbeat from core client for 30 sec - exiting 15:20:40 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:44:50 (5500): No heartbeat from core client for 30 sec - exiting 17:44:51 (5500): No heartbeat from core client for 30 sec - exiting 17:44:52 (5500): No heartbeat from core client for 30 sec - exiting 17:44:53 (5500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:54 (5500): No heartbeat from core client for 30 sec - exiting 17:44:55 (5500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:05:17 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:41 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771D3AC3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_n4on_1920_40_008321636/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Oct 2013 01:42:30	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	777,600	1,328,004	1.7078
24 Oct 2013 00:34:46	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	751,680	1,287,612	1.7130
12 Oct 2013 17:07:06	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	725,760	1,245,820	1.7166
05 Oct 2013 01:30:54	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	699,840	1,200,973	1.7161
28 Sep 2013 06:48:52	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	673,920	1,152,875	1.7107
21 Sep 2013 23:52:37	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	648,000	1,103,845	1.7035
17 Sep 2013 02:26:11	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	622,080	1,055,336	1.6965
14 Sep 2013 21:06:05	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	596,160	1,006,731	1.6887
11 Sep 2013 22:40:38	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	570,240	958,331	1.6806
02 Sep 2013 19:19:41	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	544,320	910,190	1.6722
01 Sep 2013 19:53:57	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	518,400	866,188	1.6709
31 Aug 2013 01:31:33	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	492,480	828,951	1.6832
27 Aug 2013 02:24:22	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	466,560	792,087	1.6977
25 Aug 2013 23:21:07	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	440,640	756,125	1.7160
24 Aug 2013 19:45:38	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	414,720	719,173	1.7341
22 Aug 2013 02:32:19	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	388,800	683,111	1.7570
15 Aug 2013 01:33:03	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	362,880	641,586	1.7680
14 Aug 2013 21:30:53	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	336,960	598,337	1.7757
14 Aug 2013 21:30:53	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	311,040	554,402	1.7824
14 Aug 2013 21:30:53	1281494	15799022	hadcm3n_n4on_1920_40_008321636_2	285,120	510,548	1.7906