Task 16046153

Name	hadcm3n_ofof_1900_40_008475442_0
Workunit	8626281
Created	27 Sep 2013, 10:37:08 UTC
Sent	27 Sep 2013, 14:32:29 UTC
Report deadline	27 Dec 2013, 21:59:40 UTC
Received	11 Nov 2013, 21:48:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1242378
Run time	9 days 21 hours 41 min 51 sec
CPU time	8 days 3 hours 14 min 46 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 10:50:59 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:52:10 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:52:27 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1572, iMonCtr=1 Model crash detected, will try to restart... 19:24:16 (4764): No heartbeat from core client for 30 sec - exiting 19:24:17 (4764): No heartbeat from core client for 30 sec - exiting 19:24:18 (4764): No heartbeat from core client for 30 sec - exiting 19:24:19 (4764): No heartbeat from core client for 30 sec - exiting 19:24:20 (4764): No heartbeat from core client for 30 sec - exiting 19:24:21 (4764): No heartbeat from core client for 30 sec - exiting 19:24:22 (4764): No heartbeat from core client for 30 sec - exiting 19:24:23 (4764): No heartbeat from core client for 30 sec - exiting 19:24:24 (4764): No heartbeat from core client for 30 sec - exiting 19:24:25 (4764): No heartbeat from core client for 30 sec - exiting 19:24:27 (4764): No heartbeat from core client for 30 sec - exiting 19:24:28 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:55:05 (5828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:10 (4148): No heartbeat from core client for 30 sec - exiting 08:01:12 (4148): No heartbeat from core client for 30 sec - exiting 08:01:13 (4148): No heartbeat from core client for 30 sec - exiting 08:01:14 (4148): No heartbeat from core client for 30 sec - exiting 08:01:15 (4148): No heartbeat from core client for 30 sec - exiting 08:01:16 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:33 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:10:38 (1992): No heartbeat from core client for 30 sec - exiting 13:10:39 (1992): No heartbeat from core client for 30 sec - exiting 13:10:40 (1992): No heartbeat from core client for 30 sec - exiting 13:10:41 (1992): No heartbeat from core client for 30 sec - exiting 13:10:42 (1992): No heartbeat from core client for 30 sec - exiting 13:10:43 (1992): No heartbeat from core client for 30 sec - exiting 13:10:44 (1992): No heartbeat from core client for 30 sec - exiting 13:10:45 (1992): No heartbeat from core client for 30 sec - exiting 13:10:46 (1992): No heartbeat from core client for 30 sec - exiting 13:10:47 (1992): No heartbeat from core client for 30 sec - exiting 13:10:48 (1992): No heartbeat from core client for 30 sec - exiting 13:10:49 (1992): No heartbeat from core client for 30 sec - exiting 13:10:50 (1992): No heartbeat from core client for 30 sec - exiting 13:10:51 (1992): No heartbeat from core client for 30 sec - exiting 13:10:52 (1992): No heartbeat from core client for 30 sec - exiting 13:10:53 (1992): No heartbeat from core client for 30 sec - exiting 13:10:54 (1992): No heartbeat from core client for 30 sec - exiting 13:10:55 (1992): No heartbeat from core client for 30 sec - exiting 13:10:56 (1992): No heartbeat from core client for 30 sec - exiting 13:10:57 (1992): No heartbeat from core client for 30 sec - exiting 13:10:58 (1992): No heartbeat from core client for 30 sec - exiting 13:10:59 (1992): No heartbeat from core client for 30 sec - exiting 13:11:00 (1992): No heartbeat from core client for 30 sec - exiting 13:11:01 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:02 (1992): No heartbeat from core client for 30 sec - exiting 13:11:03 (1992): No heartbeat from core client for 30 sec - exiting 13:11:04 (1992): No heartbeat from core client for 30 sec - exiting 13:11:05 (1992): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 07:58:22 (4556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:53:42 (6076): No heartbeat from core client for 30 sec - exiting 13:53:43 (6076): No heartbeat from core client for 30 sec - exiting 13:53:44 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:48:42 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:18 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:24 (4636): No heartbeat from core client for 30 sec - exiting 13:01:28 (4636): No heartbeat from core client for 30 sec - exiting 13:01:29 (4636): No heartbeat from core client for 30 sec - exiting 13:01:30 (4636): No heartbeat from core client for 30 sec - exiting 13:01:32 (4636): No heartbeat from core client for 30 sec - exiting 13:01:33 (4636): No heartbeat from core client for 30 sec - exiting 13:01:34 (4636): No heartbeat from core client for 30 sec - exiting 13:01:35 (4636): No heartbeat from core client for 30 sec - exiting 13:01:36 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:51:52 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:54 (4752): No heartbeat from core client for 30 sec - exiting 19:42:55 (4752): No heartbeat from core client for 30 sec - exiting 19:42:56 (4752): No heartbeat from core client for 30 sec - exiting 19:42:57 (4752): No heartbeat from core client for 30 sec - exiting 19:42:58 (4752): No heartbeat from core client for 30 sec - exiting 19:42:59 (4752): No heartbeat from core client for 30 sec - exiting 19:43:00 (4752): No heartbeat from core client for 30 sec - exiting 19:43:01 (4752): No heartbeat from core client for 30 sec - exiting 19:43:02 (4752): No heartbeat from core client for 30 sec - exiting 19:43:04 (4752): No heartbeat from core client for 30 sec - exiting 19:43:05 (4752): No heartbeat from core client for 30 sec - exiting 19:43:06 (4752): No heartbeat from core client for 30 sec - exiting 19:43:07 (4752): No heartbeat from core client for 30 sec - exiting 19:43:08 (4752): No heartbeat from core client for 30 sec - exiting 19:43:09 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:54:49 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:50 (4740): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:57:57 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:04:12 (4256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:56:04 (3828): No heartbeat from core client for 30 sec - exiting 18:56:05 (3828): No heartbeat from core client for 30 sec - exiting 18:56:06 (3828): No heartbeat from core client for 30 sec - exiting 18:56:07 (3828): No heartbeat from core client for 30 sec - exiting 18:56:08 (3828): No heartbeat from core client for 30 sec - exiting 18:56:09 (3828): No heartbeat from core client for 30 sec - exiting 18:56:10 (3828): No heartbeat from core client for 30 sec - exiting 18:56:11 (3828): No heartbeat from core client for 30 sec - exiting 18:56:12 (3828): No heartbeat from core client for 30 sec - exiting 18:56:13 (3828): No heartbeat from core client for 30 sec - exiting 18:56:14 (3828): No heartbeat from core client for 30 sec - exiting 18:56:16 (3828): No heartbeat from core client for 30 sec - exiting 18:56:17 (3828): No heartbeat from core client for 30 sec - exiting 18:56:18 (3828): No heartbeat from core client for 30 sec - exiting 18:56:19 (3828): No heartbeat from core client for 30 sec - exiting 18:56:20 (3828): No heartbeat from core client for 30 sec - exiting 18:56:21 (3828): No heartbeat from core client for 30 sec - exiting 18:56:22 (3828): No heartbeat from core client for 30 sec - exiting 18:56:23 (3828): No heartbeat from core client for 30 sec - exiting 18:56:24 (3828): No heartbeat from core client for 30 sec - exiting 18:56:25 (3828): No heartbeat from core client for 30 sec - exiting 18:56:27 (3828): No heartbeat from core client for 30 sec - exiting 18:56:28 (3828): No heartbeat from core client for 30 sec - exiting 18:56:29 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:30 (3828): No heartbeat from core client for 30 sec - exiting 07:25:17 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:04:39 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:53:54 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:16:03 (2732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:07:51 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:37:38 (5636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:22 (5024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:06:55 (2652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:52 (6036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:08:06 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:48:35 (4472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:51:59 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:12 (5140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:48:21 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:02:56 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:57 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:58 (6012): No heartbeat from core client for 30 sec - exiting 07:12:59 (6012): No heartbeat from core client for 30 sec - exiting 13:17:15 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4804, iMonCtr=1 Model crash detected, will try to restart... 12:54:34 (4616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:53:26 (4548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:28 (4576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:02:34 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:23:31 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:05:43 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:28:18 (4980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:29:52 (3012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:11:22 (4784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:32:19 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4180, iMonCtr=1 Model crash detected, will try to restart... 12:54:03 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:51:25 (1276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:28:30 (3472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:44:03 (1200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:41 (4544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:06:02 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:39:30 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:33:01 (5756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:28 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:38 (6052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:48:03 (5948): No heartbeat from core client for 30 sec - exiting 21:48:04 (5948): No heartbeat from core client for 30 sec - exiting 21:48:05 (5948): No heartbeat from core client for 30 sec - exiting 21:48:06 (5948): No heartbeat from core client for 30 sec - exiting 21:48:08 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:48 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:22 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:57:59 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:19 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:25:52 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x770EFF6B write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774B7383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ofof_1900_40_008475442/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Nov 2013 16:46:49	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	518,400	692,497	1.3358
07 Nov 2013 15:20:11	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	492,480	658,131	1.3364
05 Nov 2013 23:24:27	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	466,560	623,616	1.3366
04 Nov 2013 21:58:52	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	440,640	588,901	1.3365
01 Nov 2013 20:23:25	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	414,720	554,577	1.3372
30 Oct 2013 22:49:47	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	388,800	520,235	1.3381
24 Oct 2013 23:28:38	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	362,880	485,872	1.3389
23 Oct 2013 21:47:55	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	336,960	451,373	1.3395
22 Oct 2013 20:41:09	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	311,040	416,583	1.3393
21 Oct 2013 17:01:04	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	285,120	381,524	1.3381
18 Oct 2013 15:26:15	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	259,200	347,382	1.3402
16 Oct 2013 21:07:33	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	233,280	313,253	1.3428
14 Oct 2013 21:03:35	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	207,360	278,872	1.3449
11 Oct 2013 18:47:57	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	181,440	244,534	1.3477
10 Oct 2013 17:37:20	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	155,520	210,537	1.3538
09 Oct 2013 03:25:30	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	129,600	175,707	1.3558
07 Oct 2013 20:46:23	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	103,680	140,510	1.3552
04 Oct 2013 16:49:50	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	77,760	105,429	1.3558
02 Oct 2013 16:16:41	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	51,840	70,309	1.3563
30 Sep 2013 20:30:02	1242378	16046153	hadcm3n_ofof_1900_40_008475442_0	25,920	35,046	1.3521