Task 15557575

Name	hadcm3n_3ien_1940_40_008259218_1
Workunit	8414342
Created	25 Jan 2013, 21:19:23 UTC
Sent	25 Jan 2013, 21:19:25 UTC
Report deadline	27 Apr 2013, 4:46:36 UTC
Received	13 Feb 2013, 20:23:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1219590
Run time	12 days 15 hours 30 min 22 sec
CPU time	10 days 8 hours 21 min 9 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=153228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4860, iMonCtr=1 Model crash detected, will try to restart... 16:56:06 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:56:07 (5808): No heartbeat from core client for 30 sec - exiting 16:56:08 (5808): No heartbeat from core client for 30 sec - exiting 16:56:09 (5808): No heartbeat from core client for 30 sec - exiting 16:56:10 (5808): No heartbeat from core client for 30 sec - exiting 16:56:11 (5808): No heartbeat from core client for 30 sec - exiting 16:56:12 (5808): No heartbeat from core client for 30 sec - exiting 16:56:13 (5808): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5544, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 01:47:15 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:45:57 (482480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:55 (599016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=655568, iMonCtr=1 Model crash detected, will try to restart... 10:16:18 (4168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38120, iMonCtr=1 Model crash detected, will try to restart... 17:32:22 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5288, iMonCtr=1 Model crash detected, will try to restart... 09:47:05 (6048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:40:09 (1188): No heartbeat from core client for 30 sec - exiting 14:40:10 (1188): No heartbeat from core client for 30 sec - exiting 14:40:11 (1188): No heartbeat from core client for 30 sec - exiting 14:40:12 (1188): No heartbeat from core client for 30 sec - exiting 14:40:13 (1188): No heartbeat from core client for 30 sec - exiting 14:40:14 (1188): No heartbeat from core client for 30 sec - exiting 14:40:15 (1188): No heartbeat from core client for 30 sec - exiting 14:40:16 (1188): No heartbeat from core client for 30 sec - exiting 14:40:17 (1188): No heartbeat from core client for 30 sec - exiting 14:40:18 (1188): No heartbeat from core client for 30 sec - exiting 14:40:19 (1188): No heartbeat from core client for 30 sec - exiting 14:40:20 (1188): No heartbeat from core client for 30 sec - exiting 14:40:21 (1188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:54 (7320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:00 (432188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:01 (432188): No heartbeat from core client for 30 sec - exiting 12:25:02 (432188): No heartbeat from core client for 30 sec - exiting 12:25:03 (432188): No heartbeat from core client for 30 sec - exiting 12:25:05 (432188): No heartbeat from core client for 30 sec - exiting 12:25:06 (432188): No heartbeat from core client for 30 sec - exiting 12:25:07 (432188): No heartbeat from core client for 30 sec - exiting 12:26:45 (542548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:18 (524712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:19 (524712): No heartbeat from core client for 30 sec - exiting 13:17:20 (524712): No heartbeat from core client for 30 sec - exiting 13:17:21 (524712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:38:34 (592248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:36 (592248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:44:57 (1289864): No heartbeat from core client for 30 sec - exiting 17:44:58 (1289864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:59 (1289864): No heartbeat from core client for 30 sec - exiting 17:45:00 (1289864): No heartbeat from core client for 30 sec - exiting 18:03:44 (1091040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:45 (1091040): No heartbeat from core client for 30 sec - exiting 18:24:46 (10220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:38 (234280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:27:40 (234280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2928, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4576, iMonCtr=1 Model crash detected, will try to restart... 14:37:51 (6024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... C15:47:50 (5916): No heartbeat from core client for 30 sec - exiting 15:47:51 (5916): No heartbeat from core client for 30 sec - exiting 15:47:52 (5916): No heartbeat from core client for 30 sec - exiting 15:47:53 (5916): No heartbeat from core client for 30 sec - exiting 15:47:54 (5916): No heartbeat from core client for 30 sec - exiting 15:47:55 (5916): No heartbeat from core client for 30 sec - exiting 15:47:56 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:47:57 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6392, iMonCtr=1 Model crash detected, will try to restart... 16:54:26 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:33 (1076): No heartbeat from core client for 30 sec - exiting 17:03:34 (1076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:18:22 (1076): No heartbeat from core client for 30 sec - exiting 17:18:23 (1076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:31 (6000): No heartbeat from core client for 30 sec - exiting 17:44:32 (6000): No heartbeat from core client for 30 sec - exiting 17:44:33 (6000): No heartbeat from core client for 30 sec - exiting 17:44:34 (6000): No heartbeat from core client for 30 sec - exiting 17:44:35 (6000): No heartbeat from core client for 30 sec - exiting 17:44:36 (6000): No heartbeat from core client for 30 sec - exiting 17:44:37 (6000): No heartbeat from core client for 30 sec - exiting 17:44:38 (6000): No heartbeat from core client for 30 sec - exiting 17:44:39 (6000): No heartbeat from core client for 30 sec - exiting 17:44:40 (6000): No heartbeat from core client for 30 sec - exiting 17:44:41 (6000): No heartbeat from core client for 30 sec - exiting 17:44:42 (6000): No heartbeat from core client for 30 sec - exiting 17:44:43 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1716, iMonCtr=1 Model crash detected, will try to restart... 17:55:03 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:05 (5492): No heartbeat from core client for 30 sec - exiting 17:56:22 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2936, iMonCtr=1 Model crash detected, will try to restart... 20:30:41 (5276): No heartbeat from core client for 30 sec - exiting 20:30:42 (5276): No heartbeat from core client for 30 sec - exiting 20:30:43 (5276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:30:44 (5276): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7108, iMonCtr=1 Model crash detected, will try to restart... 20:53:46 (5360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:33 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3036, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77505EAB read attempt to address 0x40851FB4 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ien_1940_40_008259218/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Feb 2013 12:40:06	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	518,400	893,492	1.7236
11 Feb 2013 14:55:39	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	492,480	848,612	1.7231
10 Feb 2013 14:40:39	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	466,560	803,738	1.7227
09 Feb 2013 19:56:24	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	440,640	758,539	1.7214
08 Feb 2013 22:03:46	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	414,720	714,585	1.7231
08 Feb 2013 00:53:56	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	388,800	669,833	1.7228
06 Feb 2013 23:59:58	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	362,880	624,626	1.7213
05 Feb 2013 01:20:19	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	336,960	580,322	1.7222
03 Feb 2013 16:16:03	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	311,040	536,080	1.7235
02 Feb 2013 18:06:48	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	285,120	490,943	1.7219
01 Feb 2013 16:31:40	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	259,200	446,623	1.7231
31 Jan 2013 22:59:38	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	233,280	402,302	1.7245
31 Jan 2013 07:50:30	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	207,360	357,977	1.7264
30 Jan 2013 16:45:41	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	181,440	313,105	1.7257
29 Jan 2013 20:56:31	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	155,520	268,727	1.7279
29 Jan 2013 05:39:03	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	129,600	224,902	1.7354
28 Jan 2013 13:30:16	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	103,680	179,060	1.7270
27 Jan 2013 22:12:21	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	77,760	133,114	1.7119
27 Jan 2013 02:39:50	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	51,840	88,147	1.7004
26 Jan 2013 11:40:20	1219590	15557575	hadcm3n_3ien_1940_40_008259218_1	25,920	43,478	1.6774