Task 13028651

Name	hadcm3n_t6ko_1940_40_007317332_1
Workunit	7514762
Created	29 Jun 2011, 7:29:38 UTC
Sent	29 Jun 2011, 7:33:28 UTC
Report deadline	28 Sep 2011, 15:00:39 UTC
Received	18 Jul 2011, 21:50:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1132273
Run time	4 days 6 hours 8 min 11 sec
CPU time	3 days 23 hours 43 min 12 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.97 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3600, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:04:15 (2016): No heartbeat from core client for 30 sec - exiting 13:04:16 (2016): No heartbeat from core client for 30 sec - exiting 13:04:17 (2016): No heartbeat from core client for 30 sec - exiting 13:04:18 (2016): No heartbeat from core client for 30 sec - exiting 13:04:19 (2016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:45:16 (3304): No heartbeat from core client for 30 sec - exiting 16:45:17 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2740, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:43:52 (1716): No heartbeat from core client for 30 sec - exiting 17:43:53 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:26:46 (3888): No heartbeat from core client for 30 sec - exiting 15:26:47 (3888): No heartbeat from core client for 30 sec - exiting 15:26:48 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:10 (6020): No heartbeat from core client for 30 sec - exiting 15:28:11 (6020): No heartbeat from core client for 30 sec - exiting 15:28:12 (6020): No heartbeat from core client for 30 sec - exiting 15:28:13 (6020): No heartbeat from core client for 30 sec - exiting 15:28:14 (6020): No heartbeat from core client for 30 sec - exiting 15:28:15 (6020): No heartbeat from core client for 30 sec - exiting 15:28:16 (6020): No heartbeat from core client for 30 sec - exiting 15:28:17 (6020): No heartbeat from core client for 30 sec - exiting 15:28:18 (6020): No heartbeat from core client for 30 sec - exiting 15:28:19 (6020): No heartbeat from core client for 30 sec - exiting 15:28:20 (6020): No heartbeat from core client for 30 sec - exiting 15:28:21 (6020): No heartbeat from core client for 30 sec - exiting 15:28:22 (6020): No heartbeat from core client for 30 sec - exiting 15:28:23 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:29:03 (1552): No heartbeat from core client for 30 sec - exiting 15:29:04 (1552): No heartbeat from core client for 30 sec - exiting 15:29:05 (1552): No heartbeat from core client for 30 sec - exiting 15:29:06 (1552): No heartbeat from core client for 30 sec - exiting 15:29:07 (1552): No heartbeat from core client for 30 sec - exiting 15:29:08 (1552): No heartbeat from core client for 30 sec - exiting 15:29:09 (1552): No heartbeat from core client for 30 sec - exiting 15:29:10 (1552): No heartbeat from core client for 30 sec - exiting 15:29:11 (1552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:31:40 (992): No heartbeat from core client for 30 sec - exiting 15:31:41 (992): No heartbeat from core client for 30 sec - exiting 15:31:42 (992): No heartbeat from core client for 30 sec - exiting 15:31:43 (992): No heartbeat from core client for 30 sec - exiting 15:31:44 (992): No heartbeat from core client for 30 sec - exiting 15:31:45 (992): No heartbeat from core client for 30 sec - exiting 15:31:46 (992): No heartbeat from core client for 30 sec - exiting 15:31:47 (992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:31:48 (992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:34:56 (5704): No heartbeat from core client for 30 sec - exiting 15:34:58 (5704): No heartbeat from core client for 30 sec - exiting 15:34:59 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2272, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77AA7353 read attempt to address 0x40DB5864 Engaging BOINC Windows Runtime Debugger... 10:15:43 (4188): Can't acquire lockfile (32) - waiting 35s No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4612, selfPID=4612, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77AA3A93 read attempt to address 0x40DB586C Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_t6ko_1940_40_007317332/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 17:17:20	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	259,200	340,691	1.3144
25 Jul 2011 14:18:17	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	233,280	306,675	1.3146
25 Jul 2011 13:01:29	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	207,360	273,110	1.3171
25 Jul 2011 13:01:28	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	181,440	238,901	1.3167
25 Jul 2011 13:01:28	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	155,520	205,191	1.3194
25 Jul 2011 13:01:28	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	129,600	171,276	1.3216
09 Jul 2011 15:37:40	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	103,680	137,157	1.3229
07 Jul 2011 15:42:10	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	77,760	103,183	1.3269
02 Jul 2011 17:19:09	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	51,840	69,061	1.3322
30 Jun 2011 17:37:50	1132273	13028651	hadcm3n_t6ko_1940_40_007317332_1	25,920	34,533	1.3323