Task 15522241

Name	hadcm3n_3k1e_1940_40_008259674_1
Workunit	8414798
Created	3 Jan 2013, 9:56:11 UTC
Sent	3 Jan 2013, 9:56:13 UTC
Report deadline	4 Apr 2013, 17:23:24 UTC
Received	27 Feb 2013, 19:04:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1489121
Run time	15 days 3 hours 30 min 7 sec
CPU time	12 days 8 hours 1 min 29 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 11:30:22 (4400): No heartbeat from core client for 30 sec - exiting 11:30:23 (4400): No heartbeat from core client for 30 sec - exiting 11:30:24 (4400): No heartbeat from core client for 30 sec - exiting 11:30:25 (4400): No heartbeat from core client for 30 sec - exiting 11:30:26 (4400): No heartbeat from core client for 30 sec - exiting 11:30:27 (4400): No heartbeat from core client for 30 sec - exiting 11:30:28 (4400): No heartbeat from core client for 30 sec - exiting 11:30:29 (4400): No heartbeat from core client for 30 sec - exiting 11:30:30 (4400): No heartbeat from core client for 30 sec - exiting 11:30:31 (4400): No heartbeat from core client for 30 sec - exiting 11:30:32 (4400): No heartbeat from core client for 30 sec - exiting 11:30:33 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:24:07 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:15 (1444): No heartbeat from core client for 30 sec - exiting 10:07:16 (1444): No heartbeat from core client for 30 sec - exiting 10:07:17 (1444): No heartbeat from core client for 30 sec - exiting 10:07:18 (1444): No heartbeat from core client for 30 sec - exiting 10:07:19 (1444): No heartbeat from core client for 30 sec - exiting 10:07:20 (1444): No heartbeat from core client for 30 sec - exiting 10:07:21 (1444): No heartbeat from core client for 30 sec - exiting 10:07:22 (1444): No heartbeat from core client for 30 sec - exiting 10:07:23 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4172, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4480, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4616, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4616, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4552, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4492, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... 11:53:35 (212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77347373 read attempt to address 0x40DFEB02 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77AB3AB3 read attempt to address 0x40DFEB0A Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\Users\rt\BOINC-DATA/projects/climateprediction.net/hadcm3n_3k1e_1940_40_008259674/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Feb 2013 18:21:54	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	518,400	1,052,876	2.0310
23 Feb 2013 17:11:13	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	492,480	1,002,260	2.0351
21 Feb 2013 18:16:37	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	466,560	951,257	2.0389
17 Feb 2013 14:25:24	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	440,640	899,058	2.0403
15 Feb 2013 20:34:56	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	414,720	847,729	2.0441
12 Feb 2013 19:11:43	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	388,800	796,023	2.0474
11 Feb 2013 18:06:42	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	362,880	743,420	2.0487
09 Feb 2013 18:59:39	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	336,960	692,700	2.0557
07 Feb 2013 19:05:37	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	311,040	638,744	2.0536
04 Feb 2013 17:26:25	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	285,120	580,251	2.0351
02 Feb 2013 14:30:35	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	259,200	525,279	2.0265
30 Jan 2013 18:51:14	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	233,280	472,760	2.0266
26 Jan 2013 21:28:13	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	207,360	419,209	2.0216
24 Jan 2013 17:11:05	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	181,440	365,228	2.0129
20 Jan 2013 13:25:28	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	155,520	313,361	2.0149
16 Jan 2013 18:37:34	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	129,600	263,096	2.0301
13 Jan 2013 14:05:15	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	103,680	210,432	2.0296
08 Jan 2013 22:27:44	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	77,760	157,188	2.0215
06 Jan 2013 10:36:31	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	51,840	104,108	2.0083
04 Jan 2013 15:14:47	1261526	15522241	hadcm3n_3k1e_1940_40_008259674_1	25,920	51,288	1.9787