Task 16014527

Name	hadcm3n_n2f0_1920_40_008395971_3
Workunit	8546830
Created	13 Sep 2013, 8:13:33 UTC
Sent	13 Sep 2013, 8:35:28 UTC
Report deadline	13 Dec 2013, 16:02:39 UTC
Received	28 Sep 2013, 10:48:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1072815
Run time	5 days 22 hours 18 min 11 sec
CPU time	5 days 16 hours 7 min 15 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 11:47:21 (220): No heartbeat from core client for 30 sec - exiting 11:47:23 (220): No heartbeat from core client for 30 sec - exiting 11:47:24 (220): No heartbeat from core client for 30 sec - exiting 11:47:25 (220): No heartbeat from core client for 30 sec - exiting 11:47:26 (220): No heartbeat from core client for 30 sec - exiting 11:47:27 (220): No heartbeat from core client for 30 sec - exiting 11:47:28 (220): No heartbeat from core client for 30 sec - exiting 11:47:29 (220): No heartbeat from core client for 30 sec - exiting 11:47:30 (220): No heartbeat from core client for 30 sec - exiting 11:47:31 (220): No heartbeat from core client for 30 sec - exiting 11:47:32 (220): No heartbeat from core client for 30 sec - exiting 11:47:33 (220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:50:24 (3732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:17:35 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4748, iMonCtr=1 Model crash detected, will try to restart... 18:40:55 (6152): No heartbeat from core client for 30 sec - exiting 18:40:56 (6152): No heartbeat from core client for 30 sec - exiting 18:40:57 (6152): No heartbeat from core client for 30 sec - exiting 18:40:58 (6152): No heartbeat from core client for 30 sec - exiting 18:40:59 (6152): No heartbeat from core client for 30 sec - exiting 18:41:00 (6152): No heartbeat from core client for 30 sec - exiting 18:41:01 (6152): No heartbeat from core client for 30 sec - exiting 18:41:02 (6152): No heartbeat from core client for 30 sec - exiting 18:41:03 (6152): No heartbeat from core client for 30 sec - exiting 18:41:04 (6152): No heartbeat from core client for 30 sec - exiting 18:41:05 (6152): No heartbeat from core client for 30 sec - exiting 18:41:06 (6152): No heartbeat from core client for 30 sec - exiting 18:41:07 (6152): No heartbeat from core client for 30 sec - exiting 18:41:08 (6152): No heartbeat from core client for 30 sec - exiting 18:41:09 (6152): No heartbeat from core client for 30 sec - exiting 18:41:10 (6152): No heartbeat from core client for 30 sec - exiting 18:41:11 (6152): No heartbeat from core client for 30 sec - exiting 18:41:13 (6152): No heartbeat from core client for 30 sec - exiting 18:41:14 (6152): No heartbeat from core client for 30 sec - exiting 18:41:15 (6152): No heartbeat from core client for 30 sec - exiting 18:41:16 (6152): No heartbeat from core client for 30 sec - exiting 18:41:17 (6152): No heartbeat from core client for 30 sec - exiting 18:41:18 (6152): No heartbeat from core client for 30 sec - exiting 18:41:19 (6152): No heartbeat from core client for 30 sec - exiting 18:41:20 (6152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:42:38 (4860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3380, iMonCtr=1 Model crash detected, will try to restart... 18:56:36 (4296): No heartbeat from core client for 30 sec - exiting 18:56:37 (4296): No heartbeat from core client for 30 sec - exiting 18:56:38 (4296): No heartbeat from core client for 30 sec - exiting 18:56:39 (4296): No heartbeat from core client for 30 sec - exiting 18:56:40 (4296): No heartbeat from core client for 30 sec - exiting 18:56:41 (4296): No heartbeat from core client for 30 sec - exiting 18:56:42 (4296): No heartbeat from core client for 30 sec - exiting 18:56:43 (4296): No heartbeat from core client for 30 sec - exiting 18:56:45 (4296): No heartbeat from core client for 30 sec - exiting 18:56:46 (4296): No heartbeat from core client for 30 sec - exiting 18:56:47 (4296): No heartbeat from core client for 30 sec - exiting 18:56:48 (4296): No heartbeat from core client for 30 sec - exiting 18:56:49 (4296): No heartbeat from core client for 30 sec - exiting 18:56:50 (4296): No heartbeat from core client for 30 sec - exiting 18:56:51 (4296): No heartbeat from core client for 30 sec - exiting 18:56:52 (4296): No heartbeat from core client for 30 sec - exiting 18:56:53 (4296): No heartbeat from core client for 30 sec - exiting 18:56:54 (4296): No heartbeat from core client for 30 sec - exiting 18:56:56 (4296): No heartbeat from core client for 30 sec - exiting 18:56:57 (4296): No heartbeat from core client for 30 sec - exiting 18:56:58 (4296): No heartbeat from core client for 30 sec - exiting 18:56:59 (4296): No heartbeat from core client for 30 sec - exiting 18:57:00 (4296): No heartbeat from core client for 30 sec - exiting 18:57:01 (4296): No heartbeat from core client for 30 sec - exiting 18:57:02 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6936, iMonCtr=1 Model crash detected, will try to restart... 18:20:40 (4464): No heartbeat from core client for 30 sec - exiting 18:20:42 (4464): No heartbeat from core client for 30 sec - exiting 18:20:43 (4464): No heartbeat from core client for 30 sec - exiting 18:20:44 (4464): No heartbeat from core client for 30 sec - exiting 18:20:45 (4464): No heartbeat from core client for 30 sec - exiting 18:20:46 (4464): No heartbeat from core client for 30 sec - exiting 18:20:47 (4464): No heartbeat from core client for 30 sec - exiting 18:20:48 (4464): No heartbeat from core client for 30 sec - exiting 18:20:49 (4464): No heartbeat from core client for 30 sec - exiting 18:20:50 (4464): No heartbeat from core client for 30 sec - exiting 18:20:51 (4464): No heartbeat from core client for 30 sec - exiting 18:20:53 (4464): No heartbeat from core client for 30 sec - exiting 18:20:54 (4464): No heartbeat from core client for 30 sec - exiting 18:20:55 (4464): No heartbeat from core client for 30 sec - exiting 18:20:56 (4464): No heartbeat from core client for 30 sec - exiting 18:20:57 (4464): No heartbeat from core client for 30 sec - exiting 18:20:58 (4464): No heartbeat from core client for 30 sec - exiting 18:20:59 (4464): No heartbeat from core client for 30 sec - exiting 18:21:00 (4464): No heartbeat from core client for 30 sec - exiting 18:21:01 (4464): No heartbeat from core client for 30 sec - exiting 18:21:02 (4464): No heartbeat from core client for 30 sec - exiting 18:21:03 (4464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:45:27 (5020): No heartbeat from core client for 30 sec - exiting 08:45:28 (5020): No heartbeat from core client for 30 sec - exiting 08:45:29 (5020): No heartbeat from core client for 30 sec - exiting 08:45:30 (5020): No heartbeat from core client for 30 sec - exiting 08:45:31 (5020): No heartbeat from core client for 30 sec - exiting 08:45:32 (5020): No heartbeat from core client for 30 sec - exiting 08:45:33 (5020): No heartbeat from core client for 30 sec - exiting 08:45:34 (5020): No heartbeat from core client for 30 sec - exiting 08:45:35 (5020): No heartbeat from core client for 30 sec - exiting 08:45:36 (5020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:16:57 (6864): No heartbeat from core client for 30 sec - exiting 09:16:58 (6864): No heartbeat from core client for 30 sec - exiting 09:16:59 (6864): No heartbeat from core client for 30 sec - exiting 09:17:00 (6864): No heartbeat from core client for 30 sec - exiting 09:17:01 (6864): No heartbeat from core client for 30 sec - exiting 09:17:02 (6864): No heartbeat from core client for 30 sec - exiting 09:17:03 (6864): No heartbeat from core client for 30 sec - exiting 09:17:04 (6864): No heartbeat from core client for 30 sec - exiting 09:17:05 (6864): No heartbeat from core client for 30 sec - exiting 09:17:06 (6864): No heartbeat from core client for 30 sec - exiting 09:17:07 (6864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:17:08 (6864): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:35:02 (4256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:39:05 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:39:41 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:56:22 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:19:14 (1640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:35:12 (5712): No heartbeat from core client for 30 sec - exiting 18:35:13 (5712): No heartbeat from core client for 30 sec - exiting 18:35:14 (5712): No heartbeat from core client for 30 sec - exiting 18:35:15 (5712): No heartbeat from core client for 30 sec - exiting 18:35:16 (5712): No heartbeat from core client for 30 sec - exiting 18:35:17 (5712): No heartbeat from core client for 30 sec - exiting 18:35:18 (5712): No heartbeat from core client for 30 sec - exiting 18:35:19 (5712): No heartbeat from core client for 30 sec - exiting 18:35:21 (5712): No heartbeat from core client for 30 sec - exiting 18:35:22 (5712): No heartbeat from core client for 30 sec - exiting 18:35:23 (5712): No heartbeat from core client for 30 sec - exiting 18:35:24 (5712): No heartbeat from core client for 30 sec - exiting 18:35:25 (5712): No heartbeat from core client for 30 sec - exiting 18:35:26 (5712): No heartbeat from core client for 30 sec - exiting 18:35:27 (5712): No heartbeat from core client for 30 sec - exiting 18:35:28 (5712): No heartbeat from core client for 30 sec - exiting 18:35:29 (5712): No heartbeat from core client for 30 sec - exiting 18:35:30 (5712): No heartbeat from core client for 30 sec - exiting 18:35:31 (5712): No heartbeat from core client for 30 sec - exiting 18:35:32 (5712): No heartbeat from core client for 30 sec - exiting 18:35:34 (5712): No heartbeat from core client for 30 sec - exiting 18:35:35 (5712): No heartbeat from core client for 30 sec - exiting 18:35:36 (5712): No heartbeat from core client for 30 sec - exiting 18:35:37 (5712): No heartbeat from core client for 30 sec - exiting 18:35:38 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:44:47 (3476): No heartbeat from core client for 30 sec - exiting 18:44:48 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:40:47 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:15:00 (5572): No heartbeat from core client for 30 sec - exiting 19:15:01 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:25:56 (1492): No heartbeat from core client for 30 sec - exiting 18:25:57 (1492): No heartbeat from core client for 30 sec - exiting 18:25:58 (1492): No heartbeat from core client for 30 sec - exiting 18:25:59 (1492): No heartbeat from core client for 30 sec - exiting 18:26:00 (1492): No heartbeat from core client for 30 sec - exiting 18:26:01 (1492): No heartbeat from core client for 30 sec - exiting 18:26:02 (1492): No heartbeat from core client for 30 sec - exiting 18:26:04 (1492): No heartbeat from core client for 30 sec - exiting 18:26:05 (1492): No heartbeat from core client for 30 sec - exiting 18:26:06 (1492): No heartbeat from core client for 30 sec - exiting 18:26:07 (1492): No heartbeat from core client for 30 sec - exiting 18:26:08 (1492): No heartbeat from core client for 30 sec - exiting 18:26:09 (1492): No heartbeat from core client for 30 sec - exiting 18:26:10 (1492): No heartbeat from core client for 30 sec - exiting 18:26:11 (1492): No heartbeat from core client for 30 sec - exiting 18:26:12 (1492): No heartbeat from core client for 30 sec - exiting 18:26:13 (1492): No heartbeat from core client for 30 sec - exiting 18:26:14 (1492): No heartbeat from core client for 30 sec - exiting 18:26:16 (1492): No heartbeat from core client for 30 sec - exiting 18:26:17 (1492): No heartbeat from core client for 30 sec - exiting 18:26:18 (1492): No heartbeat from core client for 30 sec - exiting 18:26:19 (1492): No heartbeat from core client for 30 sec - exiting 18:26:20 (1492): No heartbeat from core client for 30 sec - exiting 18:26:21 (1492): No heartbeat from core client for 30 sec - exiting 18:26:22 (1492): No heartbeat from core client for 30 sec - exiting 18:26:23 (1492): No heartbeat from core client for 30 sec - exiting 18:26:24 (1492): No heartbeat from core client for 30 sec - exiting 18:26:25 (1492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:35:30 (1396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:36:04 (6692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77343AC3 read attempt to address 0x403B4943 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77343AC3 read attempt to address 0x403B4943 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_n2f0_1920_40_008395971/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Sep 2013 22:15:16	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	259,200	488,056	1.8829
26 Sep 2013 21:58:41	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	233,280	439,021	1.8819
25 Sep 2013 09:23:39	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	207,360	389,880	1.8802
23 Sep 2013 16:40:36	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	181,440	341,499	1.8822
21 Sep 2013 15:27:12	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	155,520	293,822	1.8893
20 Sep 2013 11:25:05	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	129,600	245,442	1.8938
18 Sep 2013 20:44:03	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	103,680	194,303	1.8741
16 Sep 2013 19:49:51	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	77,760	145,796	1.8749
14 Sep 2013 22:56:27	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	51,840	97,796	1.8865
13 Sep 2013 23:49:34	1072815	16014527	hadcm3n_n2f0_1920_40_008395971_3	25,920	50,277	1.9397