Task 16010527

Name	hadcm3n_37rg_2020_40_008406666_2
Workunit	8557522
Created	10 Sep 2013, 10:12:06 UTC
Sent	10 Sep 2013, 10:42:21 UTC
Report deadline	10 Dec 2013, 18:09:32 UTC
Received	4 Oct 2013, 9:22:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1272564
Run time	20 days 3 hours 24 min 38 sec
CPU time	6 days 1 hours 55 min 32 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.74 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:44:23 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 37rgko.dam44m0 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17688, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4760, iMonCtr=1 Model crash detected, will try to restart... 13:20:40 (5540): No heartbeat from core client for 30 sec - exiting 13:20:41 (5540): No heartbeat from core client for 30 sec - exiting 13:20:42 (5540): No heartbeat from core client for 30 sec - exiting 13:20:43 (5540): No heartbeat from core client for 30 sec - exiting 13:20:44 (5540): No heartbeat from core client for 30 sec - exiting 13:20:45 (5540): No heartbeat from core client for 30 sec - exiting 13:20:46 (5540): No heartbeat from core client for 30 sec - exiting 13:20:47 (5540): No heartbeat from core client for 30 sec - exiting 13:20:48 (5540): No heartbeat from core client for 30 sec - exiting 13:20:49 (5540): No heartbeat from core client for 30 sec - exiting 13:20:50 (5540): No heartbeat from core client for 30 sec - exiting 13:20:51 (5540): No heartbeat from core client for 30 sec - exiting 13:20:52 (5540): No heartbeat from core client for 30 sec - exiting 13:20:53 (5540): No heartbeat from core client for 30 sec - exiting 13:20:54 (5540): No heartbeat from core client for 30 sec - exiting 13:20:55 (5540): No heartbeat from core client for 30 sec - exiting 13:20:56 (5540): No heartbeat from core client for 30 sec - exiting 13:20:57 (5540): No heartbeat from core client for 30 sec - exiting 13:20:58 (5540): No heartbeat from core client for 30 sec - exiting 13:20:59 (5540): No heartbeat from core client for 30 sec - exiting 13:21:00 (5540): No heartbeat from core client for 30 sec - exiting 13:21:01 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:09:52 (25064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:31 (1716): No heartbeat from core client for 30 sec - exiting 19:11:32 (1716): No heartbeat from core client for 30 sec - exiting 19:11:33 (1716): No heartbeat from core client for 30 sec - exiting 19:11:34 (1716): No heartbeat from core client for 30 sec - exiting 19:11:35 (1716): No heartbeat from core client for 30 sec - exiting 19:11:36 (1716): No heartbeat from core client for 30 sec - exiting 19:11:37 (1716): No heartbeat from core client for 30 sec - exiting 19:11:38 (1716): No heartbeat from core client for 30 sec - exiting 19:11:39 (1716): No heartbeat from core client for 30 sec - exiting 19:11:40 (1716): No heartbeat from core client for 30 sec - exiting 19:11:41 (1716): No heartbeat from core client for 30 sec - exiting 19:11:42 (1716): No heartbeat from core client for 30 sec - exiting 19:11:43 (1716): No heartbeat from core client for 30 sec - exiting 19:11:44 (1716): No heartbeat from core client for 30 sec - exiting 19:11:45 (1716): No heartbeat from core client for 30 sec - exiting 19:11:46 (1716): No heartbeat from core client for 30 sec - exiting 19:11:47 (1716): No heartbeat from core client for 30 sec - exiting 19:11:48 (1716): No heartbeat from core client for 30 sec - exiting 19:11:49 (1716): No heartbeat from core client for 30 sec - exiting 19:11:50 (1716): No heartbeat from core client for 30 sec - exiting 19:11:51 (1716): No heartbeat from core client for 30 sec - exiting 19:11:52 (1716): No heartbeat from core client for 30 sec - exiting 19:11:53 (1716): No heartbeat from core client for 30 sec - exiting 19:11:54 (1716): No heartbeat from core client for 30 sec - exiting 19:11:55 (1716): No heartbeat from core client for 30 sec - exiting 19:11:56 (1716): No heartbeat from core client for 30 sec - exiting 19:11:57 (1716): No heartbeat from core client for 30 sec - exiting 19:11:58 (1716): No heartbeat from core client for 30 sec - exiting 19:11:59 (1716): No heartbeat from core client for 30 sec - exiting 19:12:00 (1716): No heartbeat from core client for 30 sec - exiting 19:12:01 (1716): No heartbeat from core client for 30 sec - exiting 19:12:02 (1716): No heartbeat from core client for 30 sec - exiting 19:12:03 (1716): No heartbeat from core client for 30 sec - exiting 19:12:04 (1716): No heartbeat from core client for 30 sec - exiting 19:12:05 (1716): No heartbeat from core client for 30 sec - exiting 19:12:06 (1716): No heartbeat from core client for 30 sec - exiting 19:12:07 (1716): No heartbeat from core client for 30 sec - exiting 19:12:08 (1716): No heartbeat from core client for 30 sec - exiting 19:12:09 (1716): No heartbeat from core client for 30 sec - exiting 19:12:11 (1716): No heartbeat from core client for 30 sec - exiting 19:12:12 (1716): No heartbeat from core client for 30 sec - exiting 19:12:13 (1716): No heartbeat from core client for 30 sec - exiting 19:12:14 (1716): No heartbeat from core client for 30 sec - exiting 19:12:15 (1716): No heartbeat from core client for 30 sec - exiting 19:12:16 (1716): No heartbeat from core client for 30 sec - exiting 19:12:17 (1716): No heartbeat from core client for 30 sec - exiting 19:12:18 (1716): No heartbeat from core client for 30 sec - exiting 19:12:19 (1716): No heartbeat from core client for 30 sec - exiting 19:12:20 (1716): No heartbeat from core client for 30 sec - exiting 19:12:21 (1716): No heartbeat from core client for 30 sec - exiting 19:12:22 (1716): No heartbeat from core client for 30 sec - exiting 19:12:23 (1716): No heartbeat from core client for 30 sec - exiting 19:12:24 (1716): No heartbeat from core client for 30 sec - exiting 19:12:25 (1716): No heartbeat from core client for 30 sec - exiting 19:12:26 (1716): No heartbeat from core client for 30 sec - exiting 19:12:27 (1716): No heartbeat from core client for 30 sec - exiting 19:12:28 (1716): No heartbeat from core client for 30 sec - exiting 19:12:30 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:31 (1716): No heartbeat from core client for 30 sec - exiting 04:12:10 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2564, iMonCtr=1 Model crash detected, will try to restart... 17:13:09 (936): No heartbeat from core client for 30 sec - exiting 17:13:11 (936): No heartbeat from core client for 30 sec - exiting 17:13:12 (936): No heartbeat from core client for 30 sec - exiting 17:13:13 (936): No heartbeat from core client for 30 sec - exiting 17:13:14 (936): No heartbeat from core client for 30 sec - exiting 17:13:15 (936): No heartbeat from core client for 30 sec - exiting 17:13:16 (936): No heartbeat from core client for 30 sec - exiting 17:13:17 (936): No heartbeat from core client for 30 sec - exiting 17:13:18 (936): No heartbeat from core client for 30 sec - exiting 17:13:19 (936): No heartbeat from core client for 30 sec - exiting 17:13:20 (936): No heartbeat from core client for 30 sec - exiting 17:13:21 (936): No heartbeat from core client for 30 sec - exiting 17:13:22 (936): No heartbeat from core client for 30 sec - exiting 17:13:23 (936): No heartbeat from core client for 30 sec - exiting 17:13:24 (936): No heartbeat from core client for 30 sec - exiting 17:13:25 (936): No heartbeat from core client for 30 sec - exiting 17:13:26 (936): No heartbeat from core client for 30 sec - exiting 17:13:27 (936): No heartbeat from core client for 30 sec - exiting 17:13:28 (936): No heartbeat from core client for 30 sec - exiting 17:13:29 (936): No heartbeat from core client for 30 sec - exiting 17:13:30 (936): No heartbeat from core client for 30 sec - exiting 17:13:31 (936): No heartbeat from core client for 30 sec - exiting 17:13:32 (936): No heartbeat from core client for 30 sec - exiting 17:13:33 (936): No heartbeat from core client for 30 sec - exiting 17:13:34 (936): No heartbeat from core client for 30 sec - exiting 17:13:35 (936): No heartbeat from core client for 30 sec - exiting 17:13:36 (936): No heartbeat from core client for 30 sec - exiting 17:13:37 (936): No heartbeat from core client for 30 sec - exiting 17:13:38 (936): No heartbeat from core client for 30 sec - exiting 17:13:39 (936): No heartbeat from core client for 30 sec - exiting 17:13:40 (936): No heartbeat from core client for 30 sec - exiting 17:13:41 (936): No heartbeat from core client for 30 sec - exiting 17:13:42 (936): No heartbeat from core client for 30 sec - exiting 17:13:43 (936): No heartbeat from core client for 30 sec - exiting 17:13:44 (936): No heartbeat from core client for 30 sec - exiting 17:13:45 (936): No heartbeat from core client for 30 sec - exiting 17:13:46 (936): No heartbeat from core client for 30 sec - exiting 17:13:48 (936): No heartbeat from core client for 30 sec - exiting 17:13:49 (936): No heartbeat from core client for 30 sec - exiting 17:13:50 (936): No heartbeat from core client for 30 sec - exiting 17:13:51 (936): No heartbeat from core client for 30 sec - exiting 17:13:52 (936): No heartbeat from core client for 30 sec - exiting 17:13:53 (936): No heartbeat from core client for 30 sec - exiting 17:13:54 (936): No heartbeat from core client for 30 sec - exiting 17:13:55 (936): No heartbeat from core client for 30 sec - exiting 17:13:56 (936): No heartbeat from core client for 30 sec - exiting 17:13:57 (936): No heartbeat from core client for 30 sec - exiting 17:13:58 (936): No heartbeat from core client for 30 sec - exiting 17:13:59 (936): No heartbeat from core client for 30 sec - exiting 17:14:00 (936): No heartbeat from core client for 30 sec - exiting 17:14:01 (936): No heartbeat from core client for 30 sec - exiting 17:14:02 (936): No heartbeat from core client for 30 sec - exiting 17:14:03 (936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:29:41 (5892): No heartbeat from core client for 30 sec - exiting 17:29:42 (5892): No heartbeat from core client for 30 sec - exiting 17:29:43 (5892): No heartbeat from core client for 30 sec - exiting 17:29:44 (5892): No heartbeat from core client for 30 sec - exiting 17:29:45 (5892): No heartbeat from core client for 30 sec - exiting 17:29:46 (5892): No heartbeat from core client for 30 sec - exiting 17:29:47 (5892): No heartbeat from core client for 30 sec - exiting 17:29:48 (5892): No heartbeat from core client for 30 sec - exiting 17:29:49 (5892): No heartbeat from core client for 30 sec - exiting 17:29:50 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:43 (4536): No heartbeat from core client for 30 sec - exiting 17:49:44 (4536): No heartbeat from core client for 30 sec - exiting 17:49:45 (4536): No heartbeat from core client for 30 sec - exiting 17:49:46 (4536): No heartbeat from core client for 30 sec - exiting 17:49:47 (4536): No heartbeat from core client for 30 sec - exiting 17:49:48 (4536): No heartbeat from core client for 30 sec - exiting 17:49:49 (4536): No heartbeat from core client for 30 sec - exiting 17:49:51 (4536): No heartbeat from core client for 30 sec - exiting 17:49:52 (4536): No heartbeat from core client for 30 sec - exiting 17:49:53 (4536): No heartbeat from core client for 30 sec - exiting 17:49:54 (4536): No heartbeat from core client for 30 sec - exiting 17:49:55 (4536): No heartbeat from core client for 30 sec - exiting 17:49:56 (4536): No heartbeat from core client for 30 sec - exiting 17:49:57 (4536): No heartbeat from core client for 30 sec - exiting 17:49:58 (4536): No heartbeat from core client for 30 sec - exiting 17:49:59 (4536): No heartbeat from core client for 30 sec - exiting 17:50:00 (4536): No heartbeat from core client for 30 sec - exiting 17:50:01 (4536): No heartbeat from core client for 30 sec - exiting 17:50:02 (4536): No heartbeat from core client for 30 sec - exiting 17:50:03 (4536): No heartbeat from core client for 30 sec - exiting 17:50:04 (4536): No heartbeat from core client for 30 sec - exiting 17:50:05 (4536): No heartbeat from core client for 30 sec - exiting 17:50:06 (4536): No heartbeat from core client for 30 sec - exiting 17:50:07 (4536): No heartbeat from core client for 30 sec - exiting 17:50:08 (4536): No heartbeat from core client for 30 sec - exiting 17:50:10 (4536): No heartbeat from core client for 30 sec - exiting 17:50:11 (4536): No heartbeat from core client for 30 sec - exiting 17:50:12 (4536): No heartbeat from core client for 30 sec - exiting 17:50:13 (4536): No heartbeat from core client for 30 sec - exiting 17:50:14 (4536): No heartbeat from core client for 30 sec - exiting 17:50:15 (4536): No heartbeat from core client for 30 sec - exiting 17:50:16 (4536): No heartbeat from core client for 30 sec - exiting 17:50:17 (4536): No heartbeat from core client for 30 sec - exiting 17:50:18 (4536): No heartbeat from core client for 30 sec - exiting 17:50:19 (4536): No heartbeat from core client for 30 sec - exiting 17:50:20 (4536): No heartbeat from core client for 30 sec - exiting 17:50:21 (4536): No heartbeat from core client for 30 sec - exiting 17:50:22 (4536): No heartbeat from core client for 30 sec - exiting 17:50:23 (4536): No heartbeat from core client for 30 sec - exiting 17:50:24 (4536): No heartbeat from core client for 30 sec - exiting 17:50:25 (4536): No heartbeat from core client for 30 sec - exiting 17:50:26 (4536): No heartbeat from core client for 30 sec - exiting 17:50:27 (4536): No heartbeat from core client for 30 sec - exiting 17:50:28 (4536): No heartbeat from core client for 30 sec - exiting 17:50:29 (4536): No heartbeat from core client for 30 sec - exiting 17:50:31 (4536): No heartbeat from core client for 30 sec - exiting 17:50:32 (4536): No heartbeat from core client for 30 sec - exiting 17:50:33 (4536): No heartbeat from core client for 30 sec - exiting 17:50:34 (4536): No heartbeat from core client for 30 sec - exiting 17:50:35 (4536): No heartbeat from core client for 30 sec - exiting 17:50:36 (4536): No heartbeat from core client for 30 sec - exiting 17:50:37 (4536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FFFF6B write attempt to address 0xC24032A9 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FE7383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_37rg_2020_40_008406666/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Oct 2013 23:56:56	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	259,200	453,670	1.7503
01 Oct 2013 19:37:50	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	233,280	386,156	1.6553
30 Sep 2013 13:43:17	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	207,360	313,203	1.5104
28 Sep 2013 15:30:52	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	181,440	262,854	1.4487
27 Sep 2013 01:09:50	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	155,520	206,577	1.3283
23 Sep 2013 09:58:27	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	129,600	167,018	1.2887
20 Sep 2013 13:56:30	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	103,680	134,236	1.2947
17 Sep 2013 14:22:53	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	77,760	99,761	1.2829
15 Sep 2013 11:09:47	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	51,840	61,851	1.1931
12 Sep 2013 23:23:59	1272564	16010527	hadcm3n_37rg_2020_40_008406666_2	25,920	29,290	1.1300