Task 15830909

Name	hadcm3n_4ank_1940_40_008311509_3
Workunit	8462644
Created	5 Jun 2013, 11:29:40 UTC
Sent	10 Jun 2013, 5:23:22 UTC
Report deadline	9 Sep 2013, 12:50:33 UTC
Received	23 Aug 2013, 17:37:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1281641
Run time	21 days 19 hours 32 min 7 sec
CPU time	19 days 19 hours 49 min 5 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.05 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1116, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5556, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:12:03 (4464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:50 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:21 (1204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1028, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:29:34 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:53 (4932): No heartbeat from core client for 30 sec - exiting 21:29:54 (4932): No heartbeat from core client for 30 sec - exiting 21:29:55 (4932): No heartbeat from core client for 30 sec - exiting 21:29:56 (4932): No heartbeat from core client for 30 sec - exiting 21:29:59 (4932): No heartbeat from core client for 30 sec - exiting 21:30:00 (4932): No heartbeat from core client for 30 sec - exiting 21:30:01 (4932): No heartbeat from core client for 30 sec - exiting 21:30:02 (4932): No heartbeat from core client for 30 sec - exiting 21:30:03 (4932): No heartbeat from core client for 30 sec - exiting 21:30:08 (4932): No heartbeat from core client for 30 sec - exiting 21:35:49 (6148): No heartbeat from core client for 30 sec - exiting 21:35:55 (6148): No heartbeat from core client for 30 sec - exiting 21:35:56 (6148): No heartbeat from core client for 30 sec - exiting 21:35:57 (6148): No heartbeat from core client for 30 sec - exiting 21:35:58 (6148): No heartbeat from core client for 30 sec - exiting 21:36:01 (6148): No heartbeat from core client for 30 sec - exiting 21:36:22 (6148): No heartbeat from core client for 30 sec - exiting 21:36:24 (6148): No heartbeat from core client for 30 sec - exiting 21:36:25 (6148): No heartbeat from core client for 30 sec - exiting 21:36:26 (6148): No heartbeat from core client for 30 sec - exiting 21:36:27 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:36:28 (6148): No heartbeat from core client for 30 sec - exiting 21:36:29 (6148): No heartbeat from core client for 30 sec - exiting 21:36:31 (6148): No heartbeat from core client for 30 sec - exiting 21:36:33 (6148): No heartbeat from core client for 30 sec - exiting 21:36:34 (6148): No heartbeat from core client for 30 sec - exiting 21:36:35 (6148): No heartbeat from core client for 30 sec - exiting 21:36:36 (6148): No heartbeat from core client for 30 sec - exiting 21:36:37 (6148): No heartbeat from core client for 30 sec - exiting 21:36:38 (6148): No heartbeat from core client for 30 sec - exiting 21:36:39 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:14:52 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:14:53 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:56:56 (3060): No heartbeat from core client for 30 sec - exiting 15:56:57 (3060): No heartbeat from core client for 30 sec - exiting 15:56:58 (3060): No heartbeat from core client for 30 sec - exiting 15:56:59 (3060): No heartbeat from core client for 30 sec - exiting 15:57:00 (3060): No heartbeat from core client for 30 sec - exiting 15:57:01 (3060): No heartbeat from core client for 30 sec - exiting 15:57:02 (3060): No heartbeat from core client for 30 sec - exiting 15:57:03 (3060): No heartbeat from core client for 30 sec - exiting 15:57:04 (3060): No heartbeat from core client for 30 sec - exiting 15:57:05 (3060): No heartbeat from core client for 30 sec - exiting 15:57:06 (3060): No heartbeat from core client for 30 sec - exiting 15:57:07 (3060): No heartbeat from core client for 30 sec - exiting 15:57:08 (3060): No heartbeat from core client for 30 sec - exiting 15:57:09 (3060): No heartbeat from core client for 30 sec - exiting 15:57:10 (3060): No heartbeat from core client for 30 sec - exiting 15:57:11 (3060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:57:12 (3060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:52:13 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:14 (5200): No heartbeat from core client for 30 sec - exiting 21:52:15 (5200): No heartbeat from core client for 30 sec - exiting 21:52:16 (5200): No heartbeat from core client for 30 sec - exiting 21:52:17 (5200): No heartbeat from core client for 30 sec - exiting 21:52:18 (5200): No heartbeat from core client for 30 sec - exiting 21:52:19 (5200): No heartbeat from core client for 30 sec - exiting 21:52:20 (5200): No heartbeat from core client for 30 sec - exiting 21:52:21 (5200): No heartbeat from core client for 30 sec - exiting 21:55:43 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:44 (1616): No heartbeat from core client for 30 sec - exiting 21:55:45 (1616): No heartbeat from core client for 30 sec - exiting 21:55:46 (1616): No heartbeat from core client for 30 sec - exiting 21:55:47 (1616): No heartbeat from core client for 30 sec - exiting 21:55:48 (1616): No heartbeat from core client for 30 sec - exiting 21:55:49 (1616): No heartbeat from core client for 30 sec - exiting 21:55:50 (1616): No heartbeat from core client for 30 sec - exiting 21:55:51 (1616): No heartbeat from core client for 30 sec - exiting 21:55:52 (1616): No heartbeat from core client for 30 sec - exiting 21:55:53 (1616): No heartbeat from core client for 30 sec - exiting 22:00:12 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:00:13 (5468): No heartbeat from core client for 30 sec - exiting 22:00:14 (5468): No heartbeat from core client for 30 sec - exiting 22:00:15 (5468): No heartbeat from core client for 30 sec - exiting 22:00:16 (5468): No heartbeat from core client for 30 sec - exiting 22:00:17 (5468): No heartbeat from core client for 30 sec - exiting 22:00:18 (5468): No heartbeat from core client for 30 sec - exiting 22:00:19 (5468): No heartbeat from core client for 30 sec - exiting 22:00:20 (5468): No heartbeat from core client for 30 sec - exiting 22:00:21 (5468): No heartbeat from core client for 30 sec - exiting 22:00:22 (5468): No heartbeat from core client for 30 sec - exiting 22:05:26 (1588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:05:27 (1588): No heartbeat from core client for 30 sec - exiting 22:05:28 (1588): No heartbeat from core client for 30 sec - exiting 22:05:29 (1588): No heartbeat from core client for 30 sec - exiting 22:05:30 (1588): No heartbeat from core client for 30 sec - exiting 22:05:31 (1588): No heartbeat from core client for 30 sec - exiting 22:05:32 (1588): No heartbeat from core client for 30 sec - exiting 22:05:33 (1588): No heartbeat from core client for 30 sec - exiting 22:05:34 (1588): No heartbeat from core client for 30 sec - exiting 22:05:35 (1588): No heartbeat from core client for 30 sec - exiting 22:05:36 (1588): No heartbeat from core client for 30 sec - exiting 22:07:35 (3440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C16:15:39 (4768): No heartbeat from core client for 30 sec - exiting 16:15:41 (4768): No heartbeat from core client for 30 sec - exiting 16:15:42 (4768): No heartbeat from core client for 30 sec - exiting 16:15:43 (4768): No heartbeat from core client for 30 sec - exiting 16:15:44 (4768): No heartbeat from core client for 30 sec - exiting 16:15:45 (4768): No heartbeat from core client for 30 sec - exiting 16:15:46 (4768): No heartbeat from core client for 30 sec - exiting 16:15:47 (4768): No heartbeat from core client for 30 sec - exiting 16:15:48 (4768): No heartbeat from core client for 30 sec - exiting 16:15:49 (4768): No heartbeat from core client for 30 sec - exiting 16:15:50 (4768): No heartbeat from core client for 30 sec - exiting 16:15:52 (4768): No heartbeat from core client for 30 sec - exiting 16:15:53 (4768): No heartbeat from core client for 30 sec - exiting 16:15:54 (4768): No heartbeat from core client for 30 sec - exiting 16:15:55 (4768): No heartbeat from core client for 30 sec - exiting 16:15:56 (4768): No heartbeat from core client for 30 sec - exiting 16:15:57 (4768): No heartbeat from core client for 30 sec - exiting 16:15:58 (4768): No heartbeat from core client for 30 sec - exiting 16:15:59 (4768): No heartbeat from core client for 30 sec - exiting 16:16:00 (4768): No heartbeat from core client for 30 sec - exiting 16:16:01 (4768): No heartbeat from core client for 30 sec - exiting 16:16:03 (4768): No heartbeat from core client for 30 sec - exiting 16:16:04 (4768): No heartbeat from core client for 30 sec - exiting 16:16:05 (4768): No heartbeat from core client for 30 sec - exiting 16:16:06 (4768): No heartbeat from core client for 30 sec - exiting 16:16:07 (4768): No heartbeat from core client for 30 sec - exiting 16:16:08 (4768): No heartbeat from core client for 30 sec - exiting 16:16:09 (4768): No heartbeat from core client for 30 sec - exiting 16:16:10 (4768): No heartbeat from core client for 30 sec - exiting 16:16:11 (4768): No heartbeat from core client for 30 sec - exiting 16:16:12 (4768): No heartbeat from core client for 30 sec - exiting 16:16:13 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4ankko.pjh6c10 Error converting file to netcdf: dataout/4ankko.pih6c10 Error converting file to netcdf: dataout/4ankko.pfh6c10 Error converting file to netcdf: dataout/4ankka.phh6c10 Error converting file to netcdf: dataout/4ankka.pgh6c10 Error converting file to netcdf: dataout/4ankka.peh6c10 Error converting file to netcdf: dataout/4ankka.pdh6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77DD7383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_4ank_1940_40_008311509/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Aug 2013 17:38:42	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	1,036,800	1,712,938	1.6521
20 Aug 2013 21:27:32	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	1,010,880	1,669,330	1.6514
20 Aug 2013 01:56:11	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	984,960	1,627,908	1.6528
17 Aug 2013 23:02:30	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	959,040	1,586,981	1.6548
15 Aug 2013 18:39:32	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	933,120	1,545,690	1.6565
14 Aug 2013 16:52:09	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	907,200	1,504,944	1.6589
14 Aug 2013 16:52:09	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	881,280	1,462,296	1.6593
14 Aug 2013 16:52:09	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	855,360	1,420,570	1.6608
14 Aug 2013 16:52:08	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	829,440	1,378,924	1.6625
14 Aug 2013 16:52:08	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	803,520	1,336,586	1.6634
14 Aug 2013 16:52:08	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	777,600	1,293,165	1.6630
14 Aug 2013 16:52:08	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	751,680	1,249,857	1.6628
14 Aug 2013 16:52:08	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	725,760	1,208,342	1.6649
26 Jul 2013 07:06:52	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	699,840	1,164,095	1.6634
25 Jul 2013 17:51:32	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	673,920	1,119,699	1.6615
24 Jul 2013 15:23:47	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	648,000	1,075,365	1.6595
23 Jul 2013 22:00:34	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	622,080	1,032,393	1.6596
23 Jul 2013 21:26:54	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	596,160	989,956	1.6606
23 Jul 2013 20:45:24	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	570,240	947,504	1.6616
23 Jul 2013 17:24:22	1281641	15830909	hadcm3n_4ank_1940_40_008311509_3	544,320	905,669	1.6639