Task 13117367

Name	hadcm3n_yifm_1900_40_007356748_0
Workunit	7554178
Created	6 Jul 2011, 14:48:34 UTC
Sent	9 Jul 2011, 10:20:15 UTC
Report deadline	8 Oct 2011, 17:47:26 UTC
Received	16 Sep 2011, 10:01:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1137733
Run time	10 days 23 hours 6 min 41 sec
CPU time	10 days 12 hours 4 min 15 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... 17:40:26 (1536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:53:16 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:59:06 (1048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:10:26 (3340): No heartbeat from core client for 30 sec - exiting 10:10:27 (3340): No heartbeat from core client for 30 sec - exiting 10:10:28 (3340): No heartbeat from core client for 30 sec - exiting 10:10:29 (3340): No heartbeat from core client for 30 sec - exiting 10:10:30 (3340): No heartbeat from core client for 30 sec - exiting 10:10:31 (3340): No heartbeat from core client for 30 sec - exiting 10:10:33 (3340): No heartbeat from core client for 30 sec - exiting 10:10:34 (3340): No heartbeat from core client for 30 sec - exiting 10:10:35 (3340): No heartbeat from core client for 30 sec - exiting 10:10:36 (3340): No heartbeat from core client for 30 sec - exiting 10:10:37 (3340): No heartbeat from core client for 30 sec - exiting 10:10:38 (3340): No heartbeat from core client for 30 sec - exiting 10:10:39 (3340): No heartbeat from core client for 30 sec - exiting 10:10:40 (3340): No heartbeat from core client for 30 sec - exiting 10:10:41 (3340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3948, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... 09:57:38 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:25 (4512): No heartbeat from core client for 30 sec - exiting 09:52:26 (4512): No heartbeat from core client for 30 sec - exiting 09:52:27 (4512): No heartbeat from core client for 30 sec - exiting 09:52:28 (4512): No heartbeat from core client for 30 sec - exiting 09:52:29 (4512): No heartbeat from core client for 30 sec - exiting 09:52:30 (4512): No heartbeat from core client for 30 sec - exiting 09:52:31 (4512): No heartbeat from core client for 30 sec - exiting 09:52:32 (4512): No heartbeat from core client for 30 sec - exiting 09:52:34 (4512): No heartbeat from core client for 30 sec - exiting 09:52:35 (4512): No heartbeat from core client for 30 sec - exiting 09:52:36 (4512): No heartbeat from core client for 30 sec - exiting 09:52:37 (4512): No heartbeat from core client for 30 sec - exiting 09:52:38 (4512): No heartbeat from core client for 30 sec - exiting 09:52:39 (4512): No heartbeat from core client for 30 sec - exiting 09:52:40 (4512): No heartbeat from core client for 30 sec - exiting 09:52:41 (4512): No heartbeat from core client for 30 sec - exiting 09:52:42 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:30:48 (3440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:39 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... 15:24:34 (4036): No heartbeat from core client for 30 sec - exiting 15:24:35 (4036): No heartbeat from core client for 30 sec - exiting 15:24:36 (4036): No heartbeat from core client for 30 sec - exiting 15:24:37 (4036): No heartbeat from core client for 30 sec - exiting 15:24:39 (4036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... 14:08:50 (4508): No heartbeat from core client for 30 sec - exiting 14:08:51 (4508): No heartbeat from core client for 30 sec - exiting 14:08:52 (4508): No heartbeat from core client for 30 sec - exiting 14:08:54 (4508): No heartbeat from core client for 30 sec - exiting 14:08:55 (4508): No heartbeat from core client for 30 sec - exiting 14:08:56 (4508): No heartbeat from core client for 30 sec - exiting 14:08:57 (4508): No heartbeat from core client for 30 sec - exiting 14:08:58 (4508): No heartbeat from core client for 30 sec - exiting 14:08:59 (4508): No heartbeat from core client for 30 sec - exiting 14:09:00 (4508): No heartbeat from core client for 30 sec - exiting 14:09:01 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4676, iMonCtr=1 Model crash detected, will try to restart... 10:42:36 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:27:39 (1772): No heartbeat from core client for 30 sec - exiting 10:27:40 (1772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:09:09 (4752): No heartbeat from core client for 30 sec - exiting 10:09:10 (4752): No heartbeat from core client for 30 sec - exiting 10:09:11 (4752): No heartbeat from core client for 30 sec - exiting 10:09:12 (4752): No heartbeat from core client for 30 sec - exiting 10:09:13 (4752): No heartbeat from core client for 30 sec - exiting 10:09:14 (4752): No heartbeat from core client for 30 sec - exiting 10:09:15 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4180, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 10:30:34 (5808): No heartbeat from core client for 30 sec - exiting 10:30:35 (5808): No heartbeat from core client for 30 sec - exiting 10:30:36 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4652, iMonCtr=1 Model crash detected, will try to restart... 10:08:16 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x773467A7 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yifm_1900_40_007356748/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Sep 2011 09:59:40	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	1,036,800	907,452	0.8752
14 Sep 2011 11:58:38	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	1,010,880	885,820	0.8763
13 Sep 2011 11:10:18	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	984,960	863,682	0.8769
12 Sep 2011 08:24:16	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	959,040	841,053	0.8770
08 Sep 2011 07:54:58	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	933,120	818,507	0.8772
08 Sep 2011 01:23:13	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	907,200	795,423	0.8768
07 Sep 2011 18:51:32	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	881,280	772,305	0.8763
07 Sep 2011 12:21:41	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	855,360	749,259	0.8760
06 Sep 2011 12:15:39	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	829,440	726,669	0.8761
05 Sep 2011 16:55:19	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	803,520	704,064	0.8762
05 Sep 2011 10:34:01	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	777,600	681,422	0.8763
02 Sep 2011 08:14:09	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	751,680	659,011	0.8767
31 Aug 2011 12:16:08	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	725,760	636,506	0.8770
30 Aug 2011 12:07:00	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	699,840	614,219	0.8777
29 Aug 2011 11:05:29	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	673,920	591,914	0.8783
24 Aug 2011 09:06:47	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	648,000	569,557	0.8789
23 Aug 2011 09:15:20	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	622,080	547,292	0.8798
22 Aug 2011 09:44:36	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	596,160	525,168	0.8809
18 Aug 2011 12:16:39	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	570,240	502,786	0.8817
17 Aug 2011 12:08:44	1137733	13117367	hadcm3n_yifm_1900_40_007356748_0	544,320	480,640	0.8830