Task 13519424

Name	hadcm3n_t1k0_1940_40_007442916_4
Workunit	7640419
Created	25 Oct 2011, 22:11:35 UTC
Sent	25 Oct 2011, 22:11:42 UTC
Report deadline	25 Jan 2012, 5:38:53 UTC
Received	3 Dec 2011, 0:50:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1038505
Run time	25 days 8 hours 4 min 11 sec
CPU time	21 days 22 hours 4 min 26 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.36 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 01:32:38 (1272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:05:28 (6784): No heartbeat from core client for 30 sec - exiting 07:05:29 (6784): No heartbeat from core client for 30 sec - exiting 07:05:30 (6784): No heartbeat from core client for 30 sec - exiting 07:05:31 (6784): No heartbeat from core client for 30 sec - exiting 07:05:32 (6784): No heartbeat from core client for 30 sec - exiting 07:05:33 (6784): No heartbeat from core client for 30 sec - exiting 07:05:34 (6784): No heartbeat from core client for 30 sec - exiting 07:05:35 (6784): No heartbeat from core client for 30 sec - exiting 07:05:36 (6784): No heartbeat from core client for 30 sec - exiting 07:05:37 (6784): No heartbeat from core client for 30 sec - exiting 07:05:39 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8588, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6568, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:22:37 (6584): No heartbeat from core client for 30 sec - exiting 19:22:38 (6584): No heartbeat from core client for 30 sec - exiting 19:22:39 (6584): No heartbeat from core client for 30 sec - exiting 19:22:40 (6584): No heartbeat from core client for 30 sec - exiting 19:22:41 (6584): No heartbeat from core client for 30 sec - exiting 19:22:42 (6584): No heartbeat from core client for 30 sec - exiting 19:22:43 (6584): No heartbeat from core client for 30 sec - exiting 19:22:44 (6584): No heartbeat from core client for 30 sec - exiting 19:22:45 (6584): No heartbeat from core client for 30 sec - exiting 19:22:46 (6584): No heartbeat from core client for 30 sec - exiting 19:22:47 (6584): No heartbeat from core client for 30 sec - exiting 19:22:48 (6584): No heartbeat from core client for 30 sec - exiting 19:22:49 (6584): No heartbeat from core client for 30 sec - exiting 19:22:50 (6584): No heartbeat from core client for 30 sec - exiting 19:22:51 (6584): No heartbeat from core client for 30 sec - exiting 19:22:52 (6584): No heartbeat from core client for 30 sec - exiting 19:22:53 (6584): No heartbeat from core client for 30 sec - exiting 19:22:54 (6584): No heartbeat from core client for 30 sec - exiting 19:22:55 (6584): No heartbeat from core client for 30 sec - exiting 19:22:56 (6584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:24:48 (6620): No heartbeat from core client for 30 sec - exiting 12:24:49 (6620): No heartbeat from core client for 30 sec - exiting 12:24:50 (6620): No heartbeat from core client for 30 sec - exiting 12:24:51 (6620): No heartbeat from core client for 30 sec - exiting 12:24:52 (6620): No heartbeat from core client for 30 sec - exiting 12:24:53 (6620): No heartbeat from core client for 30 sec - exiting 12:24:54 (6620): No heartbeat from core client for 30 sec - exiting 12:24:55 (6620): No heartbeat from core client for 30 sec - exiting 12:24:56 (6620): No heartbeat from core client for 30 sec - exiting 12:24:57 (6620): No heartbeat from core client for 30 sec - exiting 12:24:58 (6620): No heartbeat from core client for 30 sec - exiting 12:24:59 (6620): No heartbeat from core client for 30 sec - exiting 12:25:00 (6620): No heartbeat from core client for 30 sec - exiting 12:25:01 (6620): No heartbeat from core client for 30 sec - exiting 12:25:02 (6620): No heartbeat from core client for 30 sec - exiting 12:25:03 (6620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:06:40 (4196): No heartbeat from core client for 30 sec - exiting 12:06:42 (4196): No heartbeat from core client for 30 sec - exiting 12:06:43 (4196): No heartbeat from core client for 30 sec - exiting 12:06:44 (4196): No heartbeat from core client for 30 sec - exiting 12:06:45 (4196): No heartbeat from core client for 30 sec - exiting 12:06:46 (4196): No heartbeat from core client for 30 sec - exiting 12:06:47 (4196): No heartbeat from core client for 30 sec - exiting 12:06:48 (4196): No heartbeat from core client for 30 sec - exiting 12:06:49 (4196): No heartbeat from core client for 30 sec - exiting 12:06:50 (4196): No heartbeat from core client for 30 sec - exiting 12:06:51 (4196): No heartbeat from core client for 30 sec - exiting 12:06:52 (4196): No heartbeat from core client for 30 sec - exiting 12:06:53 (4196): No heartbeat from core client for 30 sec - exiting 12:06:54 (4196): No heartbeat from core client for 30 sec - exiting 12:06:55 (4196): No heartbeat from core client for 30 sec - exiting 12:06:56 (4196): No heartbeat from core client for 30 sec - exiting 12:06:57 (4196): No heartbeat from core client for 30 sec - exiting 12:06:58 (4196): No heartbeat from core client for 30 sec - exiting 12:06:59 (4196): No heartbeat from core client for 30 sec - exiting 12:07:00 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:28:57 (5720): No heartbeat from core client for 30 sec - exiting 21:28:59 (5720): No heartbeat from core client for 30 sec - exiting 21:29:00 (5720): No heartbeat from core client for 30 sec - exiting 21:29:01 (5720): No heartbeat from core client for 30 sec - exiting 21:29:02 (5720): No heartbeat from core client for 30 sec - exiting 21:29:03 (5720): No heartbeat from core client for 30 sec - exiting 21:29:04 (5720): No heartbeat from core client for 30 sec - exiting 21:29:05 (5720): No heartbeat from core client for 30 sec - exiting 21:29:06 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:21 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:28:13 (5472): No heartbeat from core client for 30 sec - exiting 00:28:14 (5472): No heartbeat from core client for 30 sec - exiting 00:28:15 (5472): No heartbeat from core client for 30 sec - exiting 00:28:16 (5472): No heartbeat from core client for 30 sec - exiting 00:28:17 (5472): No heartbeat from core client for 30 sec - exiting 00:28:18 (5472): No heartbeat from core client for 30 sec - exiting 00:28:19 (5472): No heartbeat from core client for 30 sec - exiting 00:28:20 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:18:05 (3504): No heartbeat from core client for 30 sec - exiting 15:18:06 (3504): No heartbeat from core client for 30 sec - exiting 15:18:07 (3504): No heartbeat from core client for 30 sec - exiting 15:18:08 (3504): No heartbeat from core client for 30 sec - exiting 15:18:09 (3504): No heartbeat from core client for 30 sec - exiting 15:18:10 (3504): No heartbeat from core client for 30 sec - exiting 15:18:11 (3504): No heartbeat from core client for 30 sec - exiting 15:18:12 (3504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:39:48 (2200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:26:44 (4320): No heartbeat from core client for 30 sec - exiting 17:26:45 (4320): No heartbeat from core client for 30 sec - exiting 17:26:46 (4320): No heartbeat from core client for 30 sec - exiting 17:26:47 (4320): No heartbeat from core client for 30 sec - exiting 17:26:48 (4320): No heartbeat from core client for 30 sec - exiting 17:26:49 (4320): No heartbeat from core client for 30 sec - exiting 17:26:50 (4320): No heartbeat from core client for 30 sec - exiting 17:26:51 (4320): No heartbeat from core client for 30 sec - exiting 17:26:52 (4320): No heartbeat from core client for 30 sec - exiting 17:26:53 (4320): No heartbeat from core client for 30 sec - exiting 17:26:54 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:41 (2652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:27 (5856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:41:59 (6952): No heartbeat from core client for 30 sec - exiting 21:42:00 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:33 (6944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... C15:04:18 (7060): No heartbeat from core client for 30 sec - exiting 15:04:19 (7060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:20 (7060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:00:30 (6452): No heartbeat from core client for 30 sec - exiting 18:00:31 (6452): No heartbeat from core client for 30 sec - exiting 18:00:32 (6452): No heartbeat from core client for 30 sec - exiting 18:00:33 (6452): No heartbeat from core client for 30 sec - exiting 18:00:34 (6452): No heartbeat from core client for 30 sec - exiting 18:00:35 (6452): No heartbeat from core client for 30 sec - exiting 18:00:36 (6452): No heartbeat from core client for 30 sec - exiting 18:00:37 (6452): No heartbeat from core client for 30 sec - exiting 18:00:38 (6452): No heartbeat from core client for 30 sec - exiting 18:00:39 (6452): No heartbeat from core client for 30 sec - exiting 18:00:40 (6452): No heartbeat from core client for 30 sec - exiting 18:00:41 (6452): No heartbeat from core client for 30 sec - exiting 18:00:42 (6452): No heartbeat from core client for 30 sec - exiting 18:00:43 (6452): No heartbeat from core client for 30 sec - exiting 18:00:44 (6452): No heartbeat from core client for 30 sec - exiting 18:00:45 (6452): No heartbeat from core client for 30 sec - exiting 18:00:46 (6452): No heartbeat from core client for 30 sec - exiting 18:00:47 (6452): No heartbeat from core client for 30 sec - exiting 18:00:48 (6452): No heartbeat from core client for 30 sec - exiting 18:00:49 (6452): No heartbeat from core client for 30 sec - exiting 18:00:50 (6452): No heartbeat from core client for 30 sec - exiting 18:00:51 (6452): No heartbeat from core client for 30 sec - exiting 18:00:52 (6452): No heartbeat from core client for 30 sec - exiting 18:00:53 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:32 (6812): No heartbeat from core client for 30 sec - exiting 22:25:33 (6812): No heartbeat from core client for 30 sec - exiting 22:25:34 (6812): No heartbeat from core client for 30 sec - exiting 22:25:35 (6812): No heartbeat from core client for 30 sec - exiting 22:25:36 (6812): No heartbeat from core client for 30 sec - exiting 22:25:37 (6812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77273A93 read attempt to address 0x40B80971 Engaging BOINC Windows Runtime Debugger... 01:16:20 (6336): No heartbeat from core client for 30 sec - exiting 01:16:21 (6336): No heartbeat from core client for 30 sec - exiting 01:16:22 (6336): No heartbeat from core client for 30 sec - exiting 01:16:23 (6336): No heartbeat from core client for 30 sec - exiting 01:16:54 (6336): No heartbeat from core client for 30 sec - exiting 01:16:55 (6336): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77153A93 read attempt to address 0x40B80971 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_t1k0_1940_40_007442916/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Nov 2011 03:19:23	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	1,036,800	1,893,184	1.8260
28 Nov 2011 12:21:37	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	1,010,880	1,848,137	1.8282
27 Nov 2011 18:43:57	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	984,960	1,798,680	1.8261
27 Nov 2011 02:08:43	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	959,040	1,752,209	1.8270
26 Nov 2011 10:04:54	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	933,120	1,706,523	1.8288
25 Nov 2011 14:12:20	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	907,200	1,660,902	1.8308
24 Nov 2011 18:55:10	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	881,280	1,616,160	1.8339
23 Nov 2011 18:56:19	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	855,360	1,570,563	1.8361
23 Nov 2011 02:56:13	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	829,440	1,525,742	1.8395
22 Nov 2011 11:09:21	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	803,520	1,480,032	1.8419
21 Nov 2011 12:39:29	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	777,600	1,435,716	1.8463
20 Nov 2011 22:15:46	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	751,680	1,391,960	1.8518
19 Nov 2011 01:51:53	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	725,760	1,344,149	1.8521
18 Nov 2011 05:08:12	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	699,840	1,295,313	1.8509
17 Nov 2011 11:44:09	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	673,920	1,246,646	1.8498
16 Nov 2011 20:04:18	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	648,000	1,198,446	1.8495
16 Nov 2011 02:52:13	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	622,080	1,165,068	1.8729
15 Nov 2011 17:39:07	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	596,160	1,119,841	1.8784
15 Nov 2011 17:39:07	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	570,240	1,072,161	1.8802
15 Nov 2011 17:39:07	1038505	13519424	hadcm3n_t1k0_1940_40_007442916_4	544,320	1,023,428	1.8802