Task 15777287

Name	hadcm3n_3fk2_1980_40_008366572_1
Workunit	8517431
Created	11 May 2013, 8:11:01 UTC
Sent	11 May 2013, 8:13:45 UTC
Report deadline	10 Aug 2013, 15:40:56 UTC
Received	18 Jul 2013, 11:47:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1118216
Run time	23 days 19 hours 20 min 25 sec
CPU time	22 days 8 hours 43 min 28 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6888, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:25:26 (5664): No heartbeat from core client for 30 sec - exiting 06:25:27 (5664): No heartbeat from core client for 30 sec - exiting 06:25:28 (5664): No heartbeat from core client for 30 sec - exiting 06:25:29 (5664): No heartbeat from core client for 30 sec - exiting 06:25:30 (5664): No heartbeat from core client for 30 sec - exiting 06:25:32 (5664): No heartbeat from core client for 30 sec - exiting 06:25:33 (5664): No heartbeat from core client for 30 sec - exiting 06:25:34 (5664): No heartbeat from core client for 30 sec - exiting 06:25:35 (5664): No heartbeat from core client for 30 sec - exiting 06:25:36 (5664): No heartbeat from core client for 30 sec - exiting 06:25:37 (5664): No heartbeat from core client for 30 sec - exiting 06:25:38 (5664): No heartbeat from core client for 30 sec - exiting 06:25:39 (5664): No heartbeat from core client for 30 sec - exiting 06:25:40 (5664): No heartbeat from core client for 30 sec - exiting 06:25:41 (5664): No heartbeat from core client for 30 sec - exiting 06:25:42 (5664): No heartbeat from core client for 30 sec - exiting 06:25:44 (5664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:35:18 (7088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6400, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3fk2ko.pjj4c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6232, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3516, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9072, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 08:20:20 (6096): No heartbeat from core client for 30 sec - exiting 08:20:21 (6096): No heartbeat from core client for 30 sec - exiting 08:20:22 (6096): No heartbeat from core client for 30 sec - exiting 08:20:23 (6096): No heartbeat from core client for 30 sec - exiting 08:20:24 (6096): No heartbeat from core client for 30 sec - exiting 08:20:26 (6096): No heartbeat from core client for 30 sec - exiting 08:20:27 (6096): No heartbeat from core client for 30 sec - exiting 08:20:28 (6096): No heartbeat from core client for 30 sec - exiting 08:20:29 (6096): No heartbeat from core client for 30 sec - exiting 08:20:30 (6096): No heartbeat from core client for 30 sec - exiting 08:20:31 (6096): No heartbeat from core client for 30 sec - exiting 08:20:32 (6096): No heartbeat from core client for 30 sec - exiting 08:20:33 (6096): No heartbeat from core client for 30 sec - exiting 08:20:34 (6096): No heartbeat from core client for 30 sec - exiting 08:20:35 (6096): No heartbeat from core client for 30 sec - exiting 08:20:36 (6096): No heartbeat from core client for 30 sec - exiting 08:20:38 (6096): No heartbeat from core client for 30 sec - exiting 08:20:39 (6096): No heartbeat from core client for 30 sec - exiting 08:20:40 (6096): No heartbeat from core client for 30 sec - exiting 08:20:41 (6096): No heartbeat from core client for 30 sec - exiting 08:20:42 (6096): No heartbeat from core client for 30 sec - exiting 08:20:43 (6096): No heartbeat from core client for 30 sec - exiting 08:20:44 (6096): No heartbeat from core client for 30 sec - exiting 08:20:45 (6096): No heartbeat from core client for 30 sec - exiting 08:20:46 (6096): No heartbeat from core client for 30 sec - exiting 08:20:47 (6096): No heartbeat from core client for 30 sec - exiting 08:20:48 (6096): No heartbeat from core client for 30 sec - exiting 08:20:50 (6096): No heartbeat from core client for 30 sec - exiting 08:20:51 (6096): No heartbeat from core client for 30 sec - exiting 08:20:52 (6096): No heartbeat from core client for 30 sec - exiting 08:20:53 (6096): No heartbeat from core client for 30 sec - exiting 08:20:54 (6096): No heartbeat from core client for 30 sec - exiting 08:20:55 (6096): No heartbeat from core client for 30 sec - exiting 08:20:56 (6096): No heartbeat from core client for 30 sec - exiting 08:20:57 (6096): No heartbeat from core client for 30 sec - exiting 08:20:58 (6096): No heartbeat from core client for 30 sec - exiting 08:20:59 (6096): No heartbeat from core client for 30 sec - exiting 08:21:00 (6096): No heartbeat from core client for 30 sec - exiting 08:21:02 (6096): No heartbeat from core client for 30 sec - exiting 08:21:03 (6096): No heartbeat from core client for 30 sec - exiting 08:21:04 (6096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:01:10 (6048): No heartbeat from core client for 30 sec - exiting 08:01:11 (6048): No heartbeat from core client for 30 sec - exiting 08:01:12 (6048): No heartbeat from core client for 30 sec - exiting 08:01:14 (6048): No heartbeat from core client for 30 sec - exiting 08:01:15 (6048): No heartbeat from core client for 30 sec - exiting 08:01:16 (6048): No heartbeat from core client for 30 sec - exiting 08:01:17 (6048): No heartbeat from core client for 30 sec - exiting 08:01:18 (6048): No heartbeat from core client for 30 sec - exiting 08:01:19 (6048): No heartbeat from core client for 30 sec - exiting 08:01:20 (6048): No heartbeat from core client for 30 sec - exiting 08:01:21 (6048): No heartbeat from core client for 30 sec - exiting 08:01:22 (6048): No heartbeat from core client for 30 sec - exiting 08:01:23 (6048): No heartbeat from core client for 30 sec - exiting 08:01:25 (6048): No heartbeat from core client for 30 sec - exiting 08:01:26 (6048): No heartbeat from core client for 30 sec - exiting 08:01:27 (6048): No heartbeat from core client for 30 sec - exiting 08:01:28 (6048): No heartbeat from core client for 30 sec - exiting 08:01:29 (6048): No heartbeat from core client for 30 sec - exiting 08:01:30 (6048): No heartbeat from core client for 30 sec - exiting 08:01:31 (6048): No heartbeat from core client for 30 sec - exiting 08:01:32 (6048): No heartbeat from core client for 30 sec - exiting 08:01:33 (6048): No heartbeat from core client for 30 sec - exiting 08:01:34 (6048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:11 (3708): No heartbeat from core client for 30 sec - exiting 08:44:12 (3708): No heartbeat from core client for 30 sec - exiting 08:44:13 (3708): No heartbeat from core client for 30 sec - exiting 08:44:14 (3708): No heartbeat from core client for 30 sec - exiting 08:44:15 (3708): No heartbeat from core client for 30 sec - exiting 08:44:16 (3708): No heartbeat from core client for 30 sec - exiting 08:44:17 (3708): No heartbeat from core client for 30 sec - exiting 08:44:18 (3708): No heartbeat from core client for 30 sec - exiting 08:44:19 (3708): No heartbeat from core client for 30 sec - exiting 08:44:20 (3708): No heartbeat from core client for 30 sec - exiting 08:44:21 (3708): No heartbeat from core client for 30 sec - exiting 08:44:22 (3708): No heartbeat from core client for 30 sec - exiting 08:44:23 (3708): No heartbeat from core client for 30 sec - exiting 08:44:24 (3708): No heartbeat from core client for 30 sec - exiting 08:44:25 (3708): No heartbeat from core client for 30 sec - exiting 08:44:26 (3708): No heartbeat from core client for 30 sec - exiting 08:44:27 (3708): No heartbeat from core client for 30 sec - exiting 08:44:28 (3708): No heartbeat from core client for 30 sec - exiting 08:44:29 (3708): No heartbeat from core client for 30 sec - exiting 08:44:30 (3708): No heartbeat from core client for 30 sec - exiting 08:44:31 (3708): No heartbeat from core client for 30 sec - exiting 08:44:32 (3708): No heartbeat from core client for 30 sec - exiting 08:44:33 (3708): No heartbeat from core client for 30 sec - exiting 08:44:34 (3708): No heartbeat from core client for 30 sec - exiting 08:44:35 (3708): No heartbeat from core client for 30 sec - exiting 08:44:36 (3708): No heartbeat from core client for 30 sec - exiting 08:44:37 (3708): No heartbeat from core client for 30 sec - exiting 08:44:38 (3708): No heartbeat from core client for 30 sec - exiting 08:44:39 (3708): No heartbeat from core client for 30 sec - exiting 08:44:40 (3708): No heartbeat from core client for 30 sec - exiting 08:44:41 (3708): No heartbeat from core client for 30 sec - exiting 08:44:42 (3708): No heartbeat from core client for 30 sec - exiting 08:44:43 (3708): No heartbeat from core client for 30 sec - exiting 08:44:44 (3708): No heartbeat from core client for 30 sec - exiting 08:44:45 (3708): No heartbeat from core client for 30 sec - exiting 08:44:46 (3708): No heartbeat from core client for 30 sec - exiting 08:44:47 (3708): No heartbeat from core client for 30 sec - exiting 08:44:48 (3708): No heartbeat from core client for 30 sec - exiting 08:44:49 (3708): No heartbeat from core client for 30 sec - exiting 08:44:50 (3708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:30:01 (4828): No heartbeat from core client for 30 sec - exiting 08:30:08 (4828): No heartbeat from core client for 30 sec - exiting 08:30:09 (4828): No heartbeat from core client for 30 sec - exiting 08:30:10 (4828): No heartbeat from core client for 30 sec - exiting 08:30:11 (4828): No heartbeat from core client for 30 sec - exiting 08:30:12 (4828): No heartbeat from core client for 30 sec - exiting 08:30:13 (4828): No heartbeat from core client for 30 sec - exiting 08:30:14 (4828): No heartbeat from core client for 30 sec - exiting 08:30:15 (4828): No heartbeat from core client for 30 sec - exiting 08:30:17 (4828): No heartbeat from core client for 30 sec - exiting 08:30:18 (4828): No heartbeat from core client for 30 sec - exiting 08:30:19 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7775331F read attempt to address 0x00000004 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3fk2_1980_40_008366572/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 15:15:43	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	1,036,800	1,932,203	1.8636
23 Jul 2013 15:15:43	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	1,010,880	1,885,612	1.8653
11 Jul 2013 15:58:11	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	984,960	1,839,534	1.8676
10 Jul 2013 14:37:07	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	959,040	1,791,990	1.8685
08 Jul 2013 20:59:33	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	933,120	1,743,536	1.8685
07 Jul 2013 09:31:08	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	907,200	1,689,237	1.8620
04 Jul 2013 15:40:16	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	881,280	1,640,818	1.8619
03 Jul 2013 07:50:06	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	855,360	1,591,740	1.8609
02 Jul 2013 11:34:32	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	829,440	1,543,562	1.8610
02 Jul 2013 10:10:32	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	803,520	1,494,452	1.8599
29 Jun 2013 02:30:23	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	777,600	1,445,653	1.8591
27 Jun 2013 16:07:39	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	751,680	1,396,775	1.8582
26 Jun 2013 16:13:31	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	725,760	1,348,514	1.8581
25 Jun 2013 15:55:24	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	699,840	1,299,022	1.8562
24 Jun 2013 16:18:46	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	673,920	1,250,111	1.8550
23 Jun 2013 14:44:18	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	648,000	1,200,881	1.8532
21 Jun 2013 19:59:16	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	622,080	1,152,590	1.8528
20 Jun 2013 11:59:21	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	596,160	1,103,265	1.8506
19 Jun 2013 12:07:56	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	570,240	1,054,350	1.8490
18 Jun 2013 11:21:45	1118216	15777287	hadcm3n_3fk2_1980_40_008366572_1	544,320	1,005,171	1.8467