Task 15686651

Name	hadcm3n_u3t5_2020_40_008337628_0
Workunit	8488489
Created	27 Mar 2013, 10:30:21 UTC
Sent	27 Mar 2013, 10:32:06 UTC
Report deadline	26 Jun 2013, 17:59:17 UTC
Received	25 Apr 2013, 7:44:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1267124
Run time	9 days 4 hours 23 min 42 sec
CPU time	8 days 11 hours 30 min 13 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.06 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:23:20 (5708): No heartbeat from core client for 30 sec - exiting 11:23:21 (5708): No heartbeat from core client for 30 sec - exiting 11:23:22 (5708): No heartbeat from core client for 30 sec - exiting 11:23:23 (5708): No heartbeat from core client for 30 sec - exiting 11:23:24 (5708): No heartbeat from core client for 30 sec - exiting 11:23:26 (5708): No heartbeat from core client for 30 sec - exiting 11:23:27 (5708): No heartbeat from core client for 30 sec - exiting 11:23:28 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:29 (5708): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:03:12 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:03:39 (5708): No heartbeat from core client for 30 sec - exiting 08:03:40 (5708): No heartbeat from core client for 30 sec - exiting 08:03:41 (5708): No heartbeat from core client for 30 sec - exiting 08:03:42 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:44:47 (5888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:41:45 (4416): No heartbeat from core client for 30 sec - exiting 07:41:46 (4416): No heartbeat from core client for 30 sec - exiting 07:41:47 (4416): No heartbeat from core client for 30 sec - exiting 07:41:48 (4416): No heartbeat from core client for 30 sec - exiting 07:41:49 (4416): No heartbeat from core client for 30 sec - exiting 07:41:50 (4416): No heartbeat from core client for 30 sec - exiting 07:41:51 (4416): No heartbeat from core client for 30 sec - exiting 07:41:53 (4416): No heartbeat from core client for 30 sec - exiting 07:41:54 (4416): No heartbeat from core client for 30 sec - exiting 07:41:55 (4416): No heartbeat from core client for 30 sec - exiting 07:41:56 (4416): No heartbeat from core client for 30 sec - exiting 07:41:57 (4416): No heartbeat from core client for 30 sec - exiting 07:41:58 (4416): No heartbeat from core client for 30 sec - exiting 07:41:59 (4416): No heartbeat from core client for 30 sec - exiting 07:42:00 (4416): No heartbeat from core client for 30 sec - exiting 07:42:01 (4416): No heartbeat from core client for 30 sec - exiting 07:42:02 (4416): No heartbeat from core client for 30 sec - exiting 07:42:03 (4416): No heartbeat from core client for 30 sec - exiting 07:42:04 (4416): No heartbeat from core client for 30 sec - exiting 07:42:05 (4416): No heartbeat from core client for 30 sec - exiting 07:42:07 (4416): No heartbeat from core client for 30 sec - exiting 07:42:08 (4416): No heartbeat from core client for 30 sec - exiting 07:42:09 (4416): No heartbeat from core client for 30 sec - exiting 07:42:10 (4416): No heartbeat from core client for 30 sec - exiting 07:42:11 (4416): No heartbeat from core client for 30 sec - exiting 07:42:12 (4416): No heartbeat from core client for 30 sec - exiting 07:42:13 (4416): No heartbeat from core client for 30 sec - exiting 07:42:14 (4416): No heartbeat from core client for 30 sec - exiting 07:42:15 (4416): No heartbeat from core client for 30 sec - exiting 07:42:16 (4416): No heartbeat from core client for 30 sec - exiting 07:42:17 (4416): No heartbeat from core client for 30 sec - exiting 07:42:18 (4416): No heartbeat from core client for 30 sec - exiting 07:42:19 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:20 (4416): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:11:58 (5792): No heartbeat from core client for 30 sec - exiting 08:11:59 (5792): No heartbeat from core client for 30 sec - exiting 08:12:00 (5792): No heartbeat from core client for 30 sec - exiting 08:12:01 (5792): No heartbeat from core client for 30 sec - exiting 08:12:02 (5792): No heartbeat from core client for 30 sec - exiting 08:12:03 (5792): No heartbeat from core client for 30 sec - exiting 08:12:04 (5792): No heartbeat from core client for 30 sec - exiting 08:12:05 (5792): No heartbeat from core client for 30 sec - exiting 08:12:06 (5792): No heartbeat from core client for 30 sec - exiting 08:12:07 (5792): No heartbeat from core client for 30 sec - exiting 08:12:08 (5792): No heartbeat from core client for 30 sec - exiting 08:12:09 (5792): No heartbeat from core client for 30 sec - exiting 08:12:10 (5792): No heartbeat from core client for 30 sec - exiting 08:12:11 (5792): No heartbeat from core client for 30 sec - exiting 08:12:12 (5792): No heartbeat from core client for 30 sec - exiting 08:12:13 (5792): No heartbeat from core client for 30 sec - exiting 08:12:14 (5792): No heartbeat from core client for 30 sec - exiting 08:12:15 (5792): No heartbeat from core client for 30 sec - exiting 08:12:16 (5792): No heartbeat from core client for 30 sec - exiting 08:12:17 (5792): No heartbeat from core client for 30 sec - exiting 08:12:18 (5792): No heartbeat from core client for 30 sec - exiting 08:12:19 (5792): No heartbeat from core client for 30 sec - exiting 08:12:20 (5792): No heartbeat from core client for 30 sec - exiting 08:12:21 (5792): No heartbeat from core client for 30 sec - exiting 08:12:22 (5792): No heartbeat from core client for 30 sec - exiting 08:12:23 (5792): No heartbeat from core client for 30 sec - exiting 08:12:24 (5792): No heartbeat from core client for 30 sec - exiting 08:12:25 (5792): No heartbeat from core client for 30 sec - exiting 08:12:26 (5792): No heartbeat from core client for 30 sec - exiting 08:12:27 (5792): No heartbeat from core client for 30 sec - exiting 08:12:28 (5792): No heartbeat from core client for 30 sec - exiting 08:12:29 (5792): No heartbeat from core client for 30 sec - exiting 08:12:30 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5384, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:49:30 (5444): No heartbeat from core client for 30 sec - exiting 07:49:31 (5444): No heartbeat from core client for 30 sec - exiting 07:49:32 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x772C6E5F read attempt to address 0x411AB34A Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77146E5F read attempt to address 0x4075A278 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_u3t5_2020_40_008337628/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Apr 2013 08:40:04	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	518,400	696,135	1.3429
23 Apr 2013 08:13:54	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	492,480	660,190	1.3405
22 Apr 2013 07:48:29	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	466,560	623,912	1.3373
20 Apr 2013 12:35:13	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	440,640	587,749	1.3339
19 Apr 2013 12:50:00	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	414,720	554,046	1.3360
17 Apr 2013 12:23:10	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	388,800	520,716	1.3393
16 Apr 2013 13:11:09	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	362,880	485,690	1.3384
15 Apr 2013 13:35:11	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	336,960	450,316	1.3364
14 Apr 2013 10:37:12	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	311,040	415,033	1.3343
12 Apr 2013 11:20:14	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	285,120	379,493	1.3310
11 Apr 2013 11:24:52	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	259,200	344,737	1.3300
10 Apr 2013 11:59:18	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	233,280	310,398	1.3306
09 Apr 2013 13:26:43	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	207,360	277,188	1.3367
08 Apr 2013 15:05:29	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	181,440	243,118	1.3399
07 Apr 2013 15:10:32	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	155,520	209,526	1.3473
06 Apr 2013 13:50:14	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	129,600	176,115	1.3589
05 Apr 2013 14:30:52	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	103,680	142,010	1.3697
04 Apr 2013 11:51:50	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	77,760	107,855	1.3870
03 Apr 2013 11:16:33	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	51,840	72,794	1.4042
02 Apr 2013 10:29:15	1267124	15686651	hadcm3n_u3t5_2020_40_008337628_0	25,920	35,975	1.3879