Task 15608426

Name	hadcm3n_zlzs_1920_40_008256364_1
Workunit	8411488
Created	14 Feb 2013, 17:39:26 UTC
Sent	14 Feb 2013, 17:39:36 UTC
Report deadline	17 May 2013, 1:06:47 UTC
Received	2 Mar 2013, 15:15:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1261977
Run time	4 days 3 hours 56 min 11 sec
CPU time	3 days 17 hours 56 min 5 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:02:32 (4456): No heartbeat from core client for 30 sec - exiting 12:02:33 (4456): No heartbeat from core client for 30 sec - exiting 12:02:34 (4456): No heartbeat from core client for 30 sec - exiting 12:02:36 (4456): No heartbeat from core client for 30 sec - exiting 12:02:37 (4456): No heartbeat from core client for 30 sec - exiting 12:02:38 (4456): No heartbeat from core client for 30 sec - exiting 12:02:39 (4456): No heartbeat from core client for 30 sec - exiting 12:02:40 (4456): No heartbeat from core client for 30 sec - exiting 12:02:41 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:42 (4456): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2652, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:49:36 (4748): No heartbeat from core client for 30 sec - exiting 15:49:37 (4748): No heartbeat from core client for 30 sec - exiting 15:49:39 (4748): No heartbeat from core client for 30 sec - exiting 15:49:40 (4748): No heartbeat from core client for 30 sec - exiting 15:49:41 (4748): No heartbeat from core client for 30 sec - exiting 15:49:42 (4748): No heartbeat from core client for 30 sec - exiting 15:49:43 (4748): No heartbeat from core client for 30 sec - exiting 15:49:44 (4748): No heartbeat from core client for 30 sec - exiting 15:49:45 (4748): No heartbeat from core client for 30 sec - exiting 15:49:46 (4748): No heartbeat from core client for 30 sec - exiting 15:49:47 (4748): No heartbeat from core client for 30 sec - exiting 15:49:48 (4748): No heartbeat from core client for 30 sec - exiting 15:49:49 (4748): No heartbeat from core client for 30 sec - exiting 15:49:51 (4748): No heartbeat from core client for 30 sec - exiting 15:49:52 (4748): No heartbeat from core client for 30 sec - exiting 15:49:53 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6688, iMonCtr=1 Model crash detected, will try to restart... 15:54:41 (4200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:31:58 (196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:59 (196): No heartbeat from core client for 30 sec - exiting 17:32:00 (196): No heartbeat from core client for 30 sec - exiting 17:32:01 (196): No heartbeat from core client for 30 sec - exiting 17:32:02 (196): No heartbeat from core client for 30 sec - exiting 17:32:40 (640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:21:31 (3596): No heartbeat from core client for 30 sec - exiting 14:21:32 (3596): No heartbeat from core client for 30 sec - exiting 14:21:33 (3596): No heartbeat from core client for 30 sec - exiting 14:21:34 (3596): No heartbeat from core client for 30 sec - exiting 14:21:35 (3596): No heartbeat from core client for 30 sec - exiting 14:21:37 (3596): No heartbeat from core client for 30 sec - exiting 14:21:38 (3596): No heartbeat from core client for 30 sec - exiting 14:21:39 (3596): No heartbeat from core client for 30 sec - exiting 14:21:40 (3596): No heartbeat from core client for 30 sec - exiting 14:21:41 (3596): No heartbeat from core client for 30 sec - exiting 14:21:42 (3596): No heartbeat from core client for 30 sec - exiting 14:21:43 (3596): No heartbeat from core client for 30 sec - exiting 14:21:44 (3596): No heartbeat from core client for 30 sec - exiting 14:21:45 (3596): No heartbeat from core client for 30 sec - exiting 14:21:46 (3596): No heartbeat from core client for 30 sec - exiting 14:21:47 (3596): No heartbeat from core client for 30 sec - exiting 14:21:48 (3596): No heartbeat from core client for 30 sec - exiting 14:21:49 (3596): No heartbeat from core client for 30 sec - exiting 14:21:50 (3596): No heartbeat from core client for 30 sec - exiting 14:21:51 (3596): No heartbeat from core client for 30 sec - exiting 14:21:53 (3596): No heartbeat from core client for 30 sec - exiting 14:21:54 (3596): No heartbeat from core client for 30 sec - exiting 14:21:55 (3596): No heartbeat from core client for 30 sec - exiting 14:21:56 (3596): No heartbeat from core client for 30 sec - exiting 14:21:57 (3596): No heartbeat from core client for 30 sec - exiting 14:21:58 (3596): No heartbeat from core client for 30 sec - exiting 14:21:59 (3596): No heartbeat from core client for 30 sec - exiting 14:22:00 (3596): No heartbeat from core client for 30 sec - exiting 14:22:01 (3596): No heartbeat from core client for 30 sec - exiting 14:22:02 (3596): No heartbeat from core client for 30 sec - exiting 14:22:03 (3596): No heartbeat from core client for 30 sec - exiting 14:22:05 (3596): No heartbeat from core client for 30 sec - exiting 14:22:06 (3596): No heartbeat from core client for 30 sec - exiting 14:22:07 (3596): No heartbeat from core client for 30 sec - exiting 14:22:08 (3596): No heartbeat from core client for 30 sec - exiting 14:22:09 (3596): No heartbeat from core client for 30 sec - exiting 14:22:10 (3596): No heartbeat from core client for 30 sec - exiting 14:22:11 (3596): No heartbeat from core client for 30 sec - exiting 14:22:12 (3596): No heartbeat from core client for 30 sec - exiting 14:22:13 (3596): No heartbeat from core client for 30 sec - exiting 14:22:14 (3596): No heartbeat from core client for 30 sec - exiting 14:22:15 (3596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:22:16 (3596): No heartbeat from core client for 30 sec - exiting 14:22:18 (3596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:11:20 (4476): No heartbeat from core client for 30 sec - exiting 13:11:21 (4476): No heartbeat from core client for 30 sec - exiting 13:11:22 (4476): No heartbeat from core client for 30 sec - exiting 13:11:23 (4476): No heartbeat from core client for 30 sec - exiting 13:11:25 (4476): No heartbeat from core client for 30 sec - exiting 13:11:26 (4476): No heartbeat from core client for 30 sec - exiting 13:11:27 (4476): No heartbeat from core client for 30 sec - exiting 13:11:28 (4476): No heartbeat from core client for 30 sec - exiting 13:11:29 (4476): No heartbeat from core client for 30 sec - exiting 13:11:30 (4476): No heartbeat from core client for 30 sec - exiting 13:11:31 (4476): No heartbeat from core client for 30 sec - exiting 13:11:32 (4476): No heartbeat from core client for 30 sec - exiting 13:11:33 (4476): No heartbeat from core client for 30 sec - exiting 13:11:34 (4476): No heartbeat from core client for 30 sec - exiting 13:11:35 (4476): No heartbeat from core client for 30 sec - exiting 13:11:36 (4476): No heartbeat from core client for 30 sec - exiting 13:11:37 (4476): No heartbeat from core client for 30 sec - exiting 13:11:38 (4476): No heartbeat from core client for 30 sec - exiting 13:11:39 (4476): No heartbeat from core client for 30 sec - exiting 13:11:40 (4476): No heartbeat from core client for 30 sec - exiting 13:11:42 (4476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:43 (4476): No heartbeat from core client for 30 sec - exiting 13:12:54 (2348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:41 (4372): No heartbeat from core client for 30 sec - exiting 13:44:42 (4372): No heartbeat from core client for 30 sec - exiting 13:44:43 (4372): No heartbeat from core client for 30 sec - exiting 13:44:44 (4372): No heartbeat from core client for 30 sec - exiting 13:44:46 (4372): No heartbeat from core client for 30 sec - exiting 13:44:47 (4372): No heartbeat from core client for 30 sec - exiting 13:44:48 (4372): No heartbeat from core client for 30 sec - exiting 13:44:49 (4372): No heartbeat from core client for 30 sec - exiting 13:44:50 (4372): No heartbeat from core client for 30 sec - exiting 13:44:51 (4372): No heartbeat from core client for 30 sec - exiting 13:44:52 (4372): No heartbeat from core client for 30 sec - exiting 13:44:53 (4372): No heartbeat from core client for 30 sec - exiting 13:44:54 (4372): No heartbeat from core client for 30 sec - exiting 13:44:55 (4372): No heartbeat from core client for 30 sec - exiting 13:44:56 (4372): No heartbeat from core client for 30 sec - exiting 13:44:57 (4372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zlzsko.pjc9c10 Error converting file to netcdf: dataout/zlzsko.pic9c10 Error converting file to netcdf: dataout/zlzsko.pfc9c10 Error converting file to netcdf: dataout/zlzska.phc9c10 Error converting file to netcdf: dataout/zlzska.pgc9c10 Error converting file to netcdf: dataout/zlzska.pec9c10 16:16:39 (4080): No heartbeat from core client for 30 sec - exiting 16:16:40 (4080): No heartbeat from core client for 30 sec - exiting 16:16:41 (4080): No heartbeat from core client for 30 sec - exiting 16:16:42 (4080): No heartbeat from core client for 30 sec - exiting 16:16:43 (4080): No heartbeat from core client for 30 sec - exiting 16:16:44 (4080): No heartbeat from core client for 30 sec - exiting 16:16:45 (4080): No heartbeat from core client for 30 sec - exiting 16:16:47 (4080): No heartbeat from core client for 30 sec - exiting 16:16:48 (4080): No heartbeat from core client for 30 sec - exiting 16:16:49 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:13:29 (4908): No heartbeat from core client for 30 sec - exiting 15:13:30 (4908): No heartbeat from core client for 30 sec - exiting 15:13:31 (4908): No heartbeat from core client for 30 sec - exiting 15:13:32 (4908): No heartbeat from core client for 30 sec - exiting 15:13:33 (4908): No heartbeat from core client for 30 sec - exiting 15:13:34 (4908): No heartbeat from core client for 30 sec - exiting 15:13:35 (4908): No heartbeat from core client for 30 sec - exiting 15:13:36 (4908): No heartbeat from core client for 30 sec - exiting 15:13:37 (4908): No heartbeat from core client for 30 sec - exiting 15:13:38 (4908): No heartbeat from core client for 30 sec - exiting 15:13:39 (4908): No heartbeat from core client for 30 sec - exiting 15:13:40 (4908): No heartbeat from core client for 30 sec - exiting 15:13:41 (4908): No heartbeat from core client for 30 sec - exiting 15:13:42 (4908): No heartbeat from core client for 30 sec - exiting 15:13:43 (4908): No heartbeat from core client for 30 sec - exiting 15:13:44 (4908): No heartbeat from core client for 30 sec - exiting 15:13:45 (4908): No heartbeat from core client for 30 sec - exiting 15:13:46 (4908): No heartbeat from core client for 30 sec - exiting 15:13:47 (4908): No heartbeat from core client for 30 sec - exiting 15:13:48 (4908): No heartbeat from core client for 30 sec - exiting 15:13:49 (4908): No heartbeat from core client for 30 sec - exiting 15:13:50 (4908): No heartbeat from core client for 30 sec - exiting 15:13:51 (4908): No heartbeat from core client for 30 sec - exiting 15:13:52 (4908): No heartbeat from core client for 30 sec - exiting 15:13:53 (4908): No heartbeat from core client for 30 sec - exiting 15:13:55 (4908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77447373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zlzs_1920_40_008256364/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Mar 2013 15:15:40	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	259,200	323,759	1.2491
27 Feb 2013 18:14:19	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	233,280	290,761	1.2464
26 Feb 2013 14:40:38	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	207,360	257,974	1.2441
23 Feb 2013 21:09:40	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	181,440	224,807	1.2390
22 Feb 2013 17:56:22	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	155,520	192,436	1.2374
20 Feb 2013 17:16:06	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	129,600	160,403	1.2377
19 Feb 2013 14:55:45	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	103,680	128,420	1.2386
17 Feb 2013 15:40:39	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	77,760	95,424	1.2272
16 Feb 2013 17:46:25	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	51,840	63,288	1.2208
15 Feb 2013 19:09:16	1261977	15608426	hadcm3n_zlzs_1920_40_008256364_1	25,920	32,057	1.2368