Task 13288099

Name	hadcm3n_p2vk_1940_40_007420179_0
Workunit	7617814
Created	24 Aug 2011, 21:36:29 UTC
Sent	24 Aug 2011, 21:36:56 UTC
Report deadline	24 Nov 2011, 5:04:07 UTC
Received	7 Jan 2012, 21:15:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1041683
Run time	9 days 16 hours 20 min 42 sec
CPU time	8 days 8 hours 26 min 17 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.16 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 19:50:15 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... 18:04:06 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3292, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3292, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3292, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3292, iMonCtr=1 Model crash detected, will try to restart... 13:34:05 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6548, iMonCtr=1 Model crash detected, will try to restart... 23:43:40 (3100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:47:13 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:18 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:02 (904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=852, iMonCtr=1 Model crash detected, will try to restart... 18:14:12 (5772): No heartbeat from core client for 30 sec - exiting 18:14:13 (5772): No heartbeat from core client for 30 sec - exiting 18:14:14 (5772): No heartbeat from core client for 30 sec - exiting 18:14:15 (5772): No heartbeat from core client for 30 sec - exiting 18:14:16 (5772): No heartbeat from core client for 30 sec - exiting 18:14:17 (5772): No heartbeat from core client for 30 sec - exiting 18:14:18 (5772): No heartbeat from core client for 30 sec - exiting 18:14:20 (5772): No heartbeat from core client for 30 sec - exiting 18:14:21 (5772): No heartbeat from core client for 30 sec - exiting 18:14:22 (5772): No heartbeat from core client for 30 sec - exiting 18:14:23 (5772): No heartbeat from core client for 30 sec - exiting 18:14:24 (5772): No heartbeat from core client for 30 sec - exiting 18:14:25 (5772): No heartbeat from core client for 30 sec - exiting 18:14:26 (5772): No heartbeat from core client for 30 sec - exiting 18:14:27 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6060, iMonCtr=1 Model crash detected, will try to restart... 22:14:18 (3976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:19 (3448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6796, iMonCtr=1 Model crash detected, will try to restart... 23:13:32 (1020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:02:53 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6784, iMonCtr=1 Model crash detected, will try to restart... 21:15:03 (2708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2040, iMonCtr=1 Model crash detected, will try to restart... 20:28:11 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:13:08 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:11 (5592): No heartbeat from core client for 30 sec - exiting 17:01:13 (5592): No heartbeat from core client for 30 sec - exiting 17:01:14 (5592): No heartbeat from core client for 30 sec - exiting 17:01:15 (5592): No heartbeat from core client for 30 sec - exiting 17:01:16 (5592): No heartbeat from core client for 30 sec - exiting 17:01:17 (5592): No heartbeat from core client for 30 sec - exiting 17:01:18 (5592): No heartbeat from core client for 30 sec - exiting 17:01:19 (5592): No heartbeat from core client for 30 sec - exiting 17:01:20 (5592): No heartbeat from core client for 30 sec - exiting 17:01:21 (5592): No heartbeat from core client for 30 sec - exiting 17:01:22 (5592): No heartbeat from core client for 30 sec - exiting 17:01:24 (5592): No heartbeat from core client for 30 sec - exiting 17:01:25 (5592): No heartbeat from core client for 30 sec - exiting 17:01:26 (5592): No heartbeat from core client for 30 sec - exiting 17:01:27 (5592): No heartbeat from core client for 30 sec - exiting 17:01:28 (5592): No heartbeat from core client for 30 sec - exiting 17:01:29 (5592): No heartbeat from core client for 30 sec - exiting 17:01:30 (5592): No heartbeat from core client for 30 sec - exiting 17:01:31 (5592): No heartbeat from core client for 30 sec - exiting 17:01:32 (5592): No heartbeat from core client for 30 sec - exiting 17:01:34 (5592): No heartbeat from core client for 30 sec - exiting 17:01:35 (5592): No heartbeat from core client for 30 sec - exiting 17:01:36 (5592): No heartbeat from core client for 30 sec - exiting 17:01:37 (5592): No heartbeat from core client for 30 sec - exiting 17:01:38 (5592): No heartbeat from core client for 30 sec - exiting 17:01:39 (5592): No heartbeat from core client for 30 sec - exiting 17:01:40 (5592): No heartbeat from core client for 30 sec - exiting 17:01:41 (5592): No heartbeat from core client for 30 sec - exiting 17:01:42 (5592): No heartbeat from core client for 30 sec - exiting 17:01:43 (5592): No heartbeat from core client for 30 sec - exiting 17:01:45 (5592): No heartbeat from core client for 30 sec - exiting 17:01:46 (5592): No heartbeat from core client for 30 sec - exiting 17:01:47 (5592): No heartbeat from core client for 30 sec - exiting 17:01:48 (5592): No heartbeat from core client for 30 sec - exiting 17:01:49 (5592): No heartbeat from core client for 30 sec - exiting 17:01:50 (5592): No heartbeat from core client for 30 sec - exiting 17:01:51 (5592): No heartbeat from core client for 30 sec - exiting 17:01:52 (5592): No heartbeat from core client for 30 sec - exiting 17:01:53 (5592): No heartbeat from core client for 30 sec - exiting 17:01:54 (5592): No heartbeat from core client for 30 sec - exiting 17:01:55 (5592): No heartbeat from core client for 30 sec - exiting 17:01:57 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:35 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:36 (6104): No heartbeat from core client for 30 sec - exiting 17:05:37 (6104): No heartbeat from core client for 30 sec - exiting 17:05:38 (6104): No heartbeat from core client for 30 sec - exiting 17:05:39 (6104): No heartbeat from core client for 30 sec - exiting 17:05:40 (6104): No heartbeat from core client for 30 sec - exiting 17:05:41 (6104): No heartbeat from core client for 30 sec - exiting 17:05:42 (6104): No heartbeat from core client for 30 sec - exiting 17:05:43 (6104): No heartbeat from core client for 30 sec - exiting 17:05:45 (6104): No heartbeat from core client for 30 sec - exiting 17:05:46 (6104): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5036, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6216, iMonCtr=1 Model crash detected, will try to restart... 18:27:46 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2148, iMonCtr=1 Model crash detected, will try to restart... 11:38:29 (5076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:02 (3636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:09:19 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3968, iMonCtr=1 Model crash detected, will try to restart... 08:12:18 (3280): No heartbeat from core client for 30 sec - exiting 08:12:19 (3280): No heartbeat from core client for 30 sec - exiting 08:12:20 (3280): No heartbeat from core client for 30 sec - exiting 08:12:21 (3280): No heartbeat from core client for 30 sec - exiting 08:12:22 (3280): No heartbeat from core client for 30 sec - exiting 08:12:23 (3280): No heartbeat from core client for 30 sec - exiting 08:12:24 (3280): No heartbeat from core client for 30 sec - exiting 08:12:25 (3280): No heartbeat from core client for 30 sec - exiting 08:12:27 (3280): No heartbeat from core client for 30 sec - exiting 08:12:28 (3280): No heartbeat from core client for 30 sec - exiting 08:12:29 (3280): No heartbeat from core client for 30 sec - exiting 08:12:30 (3280): No heartbeat from core client for 30 sec - exiting 08:12:31 (3280): No heartbeat from core client for 30 sec - exiting 08:12:32 (3280): No heartbeat from core client for 30 sec - exiting 08:12:33 (3280): No heartbeat from core client for 30 sec - exiting 08:12:34 (3280): No heartbeat from core client for 30 sec - exiting 08:12:35 (3280): No heartbeat from core client for 30 sec - exiting 08:12:36 (3280): No heartbeat from core client for 30 sec - exiting 08:12:38 (3280): No heartbeat from core client for 30 sec - exiting 08:12:39 (3280): No heartbeat from core client for 30 sec - exiting 08:12:40 (3280): No heartbeat from core client for 30 sec - exiting 08:12:41 (3280): No heartbeat from core client for 30 sec - exiting 08:12:42 (3280): No heartbeat from core client for 30 sec - exiting 08:12:43 (3280): No heartbeat from core client for 30 sec - exiting 08:12:44 (3280): No heartbeat from core client for 30 sec - exiting 08:12:45 (3280): No heartbeat from core client for 30 sec - exiting 08:12:46 (3280): No heartbeat from core client for 30 sec - exiting 08:12:47 (3280): No heartbeat from core client for 30 sec - exiting 08:12:48 (3280): No heartbeat from core client for 30 sec - exiting 08:12:50 (3280): No heartbeat from core client for 30 sec - exiting 08:12:51 (3280): No heartbeat from core client for 30 sec - exiting 08:12:52 (3280): No heartbeat from core client for 30 sec - exiting 08:12:53 (3280): No heartbeat from core client for 30 sec - exiting 08:12:54 (3280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:55 (3280): No heartbeat from core client for 30 sec - exiting 08:12:56 (3280): No heartbeat from core client for 30 sec - exiting 08:12:57 (3280): No heartbeat from core client for 30 sec - exiting 08:12:58 (3280): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6308, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6612, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on p2vkko.daf47u0 13:07:53 (3780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:09:37 (4988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:05:33 (6020): No heartbeat from core client for 30 sec - exiting 13:05:34 (6020): No heartbeat from core client for 30 sec - exiting 13:05:35 (6020): No heartbeat from core client for 30 sec - exiting 13:05:37 (6020): No heartbeat from core client for 30 sec - exiting 13:05:38 (6020): No heartbeat from core client for 30 sec - exiting 13:05:39 (6020): No heartbeat from core client for 30 sec - exiting 13:05:40 (6020): No heartbeat from core client for 30 sec - exiting 13:05:41 (6020): No heartbeat from core client for 30 sec - exiting 13:05:42 (6020): No heartbeat from core client for 30 sec - exiting 13:05:43 (6020): No heartbeat from core client for 30 sec - exiting 13:05:44 (6020): No heartbeat from core client for 30 sec - exiting 13:05:45 (6020): No heartbeat from core client for 30 sec - exiting 13:05:46 (6020): No heartbeat from core client for 30 sec - exiting 13:05:47 (6020): No heartbeat from core client for 30 sec - exiting 13:05:49 (6020): No heartbeat from core client for 30 sec - exiting 13:05:50 (6020): No heartbeat from core client for 30 sec - exiting 13:05:51 (6020): No heartbeat from core client for 30 sec - exiting 13:05:52 (6020): No heartbeat from core client for 30 sec - exiting 13:05:53 (6020): No heartbeat from core client for 30 sec - exiting 13:05:54 (6020): No heartbeat from core client for 30 sec - exiting 13:05:55 (6020): No heartbeat from core client for 30 sec - exiting 13:05:56 (6020): No heartbeat from core client for 30 sec - exiting 13:05:57 (6020): No heartbeat from core client for 30 sec - exiting 13:05:58 (6020): No heartbeat from core client for 30 sec - exiting 13:05:59 (6020): No heartbeat from core client for 30 sec - exiting 13:06:01 (6020): No heartbeat from core client for 30 sec - exiting 13:06:02 (6020): No heartbeat from core client for 30 sec - exiting 13:06:03 (6020): No heartbeat from core client for 30 sec - exiting 13:06:04 (6020): No heartbeat from core client for 30 sec - exiting 13:06:05 (6020): No heartbeat from core client for 30 sec - exiting 13:06:06 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6028, iMonCtr=1 Model crash detected, will try to restart... 22:39:58 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:40:00 (3720): No heartbeat from core client for 30 sec - exiting 22:40:01 (3720): No heartbeat from core client for 30 sec - exiting 22:40:02 (3720): No heartbeat from core client for 30 sec - exiting 22:40:03 (3720): No heartbeat from core client for 30 sec - exiting 22:40:04 (3720): No heartbeat from core client for 30 sec - exiting 22:40:05 (3720): No heartbeat from core client for 30 sec - exiting 22:40:06 (3720): No heartbeat from core client for 30 sec - exiting 22:40:07 (3720): No heartbeat from core client for 30 sec - exiting 22:40:09 (3720): No heartbeat from core client for 30 sec - exiting 22:42:42 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FC3A93 read attempt to address 0x40B172E5 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x00417B59 read attempt to address 0x80000004 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_p2vk_1940_40_007420179/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jan 2012 17:59:20	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	518,400	712,153	1.3738
04 Jan 2012 00:18:58	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	492,480	676,930	1.3745
02 Jan 2012 01:33:17	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	466,560	641,347	1.3746
31 Dec 2011 03:04:59	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	440,640	605,087	1.3732
29 Dec 2011 00:36:32	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	414,720	569,401	1.3730
26 Dec 2011 06:09:39	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	388,800	534,590	1.3750
25 Dec 2011 17:40:34	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	362,880	499,690	1.3770
05 Dec 2011 01:14:09	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	336,960	463,949	1.3769
29 Nov 2011 02:44:07	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	311,040	428,805	1.3786
26 Nov 2011 19:41:06	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	285,120	393,417	1.3798
16 Nov 2011 23:45:59	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	259,200	356,764	1.3764
16 Nov 2011 23:45:59	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	233,280	319,114	1.3679
08 Nov 2011 02:43:41	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	207,360	281,156	1.3559
01 Nov 2011 01:02:32	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	181,440	245,587	1.3535
31 Oct 2011 15:26:03	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	155,520	209,181	1.3450
13 Oct 2011 04:49:58	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	129,600	174,070	1.3431
02 Oct 2011 18:43:28	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	103,680	139,650	1.3469
25 Sep 2011 23:33:40	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	77,760	104,228	1.3404
11 Sep 2011 21:15:29	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	51,840	69,370	1.3382
03 Sep 2011 02:31:54	1041683	13288099	hadcm3n_p2vk_1940_40_007420179_0	25,920	34,327	1.3243