Task 15700053

Name	hadcm3n_n04z_1960_40_008341239_1
Workunit	8492100
Created	2 Apr 2013, 19:03:25 UTC
Sent	2 Apr 2013, 19:03:28 UTC
Report deadline	3 Jul 2013, 2:30:39 UTC
Received	8 Apr 2013, 15:16:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1274005
Run time	3 days 18 hours 51 min 46 sec
CPU time	3 days 17 hours 23 min 45 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 15:34:56 (9180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:35:01 (9180): No heartbeat from core client for 30 sec - exiting 15:35:02 (9180): No heartbeat from core client for 30 sec - exiting 15:35:03 (9180): No heartbeat from core client for 30 sec - exiting 15:35:05 (9180): No heartbeat from core client for 30 sec - exiting 15:35:06 (9180): No heartbeat from core client for 30 sec - exiting 15:35:07 (9180): No heartbeat from core client for 30 sec - exiting 15:35:13 (9180): No heartbeat from core client for 30 sec - exiting 15:35:15 (9180): No heartbeat from core client for 30 sec - exiting 15:35:19 (9180): No heartbeat from core client for 30 sec - exiting 15:35:25 (9180):16:08:50 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:57 (3532): No heartbeat from core client for 30 sec - exiting 16:08:58 (3532): No heartbeat from core client for 30 sec - exiting 16:09:04 (3532): No heartbeat from core client for 30 sec - exiting 16:09:08 (3532): No heartbeat from core client for 30 sec - exiting 16:09:09 (3532): No heartbeat from core client for 30 sec - exiting 16:09:17 (3532): No heartbeat from core client for 30 sec - exiting 16:55:52 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:55:20 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:55:22 (5324): No heartbeat from core client for 30 sec - exiting 02:59:37 (2120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:39 (2120): No heartbeat from core client for 30 sec - exiting 02:59:40 (2120): No heartbeat from core client for 30 sec - exiting 02:59:41 (2120): No heartbeat from core client for 30 sec - exiting 03:46:05 (1840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:16 (1840): No heartbeat from core client for 30 sec - exiting 03:46:25 (1840): No heartbeat from core client for 30 sec - exiting 03:46:33 (1840): No heartbeat from core client for 30 sec - exiting 03:46:34 (1840): No heartbeat from core client for 30 sec - exiting 03:46:35 (1840): No heartbeat from core client for 30 sec - exiting 03:46:36 (1840): No heartbeat from core client for 30 sec - exiting 03:46:37 (1840): No heartbeat from core client for 30 sec - exiting 04:36:32 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:34 (4944): No heartbeat from core client for 30 sec - exiting 05:22:58 (6808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:11 (6808): No heartbeat from core client for 30 sec - exiting 05:23:14 (6808): No heartbeat from core client for 30 sec - exiting 05:23:17 (6808):06:12:45 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:12:46 (2148): No heartbeat from core client for 30 sec - exiting 06:58:44 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:58:50 (4568): No heartbeat from core client for 30 sec - exiting 06:58:56 (4568): No heartbeat from core client for 30 sec - exiting 06:59:05 (4568): No heartbeat from core client for 30 sec - exiting 06:59:08 (4568): No heartbeat from core client for 30 sec - exiting 06:59:16 (4568): No heartbeat from core client for 30 sec - exiting 06:59:27 (4568): No heartbeat from core client for 30 sec - exiting 06:59:32 (4568): No heartbeat from core client for 30 sec - exiting 06:59:40 (4568): No heartbeat from core client for 30 sec - exiting 06:59:43 (4568):CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:47:45 (5756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:47:49 (5756): No heartbeat from core client for 30 sec - exiting 11:47:50 (5756): No heartbeat from core client for 30 sec - exiting 11:47:51 (5756): No heartbeat from core client for 30 sec - exiting 11:47:52 (5756): No heartbeat from core client for 30 sec - exiting 11:47:53 (5756): No heartbeat from core client for 30 sec - exiting 11:47:54 (5756): No heartbeat from core client for 30 sec - exiting 11:47:55 (5756): No heartbeat from core client for 30 sec - exiting 12:36:08 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:07 (4692): No heartbeat from core client for 30 sec - exiting 12:37:18 (4692): No heartbeat from core client for 30 sec - exiting 12:37:32 (4692): No heartbeat from core client for 30 sec - exiting 12:37:42 (4692): No heartbeat from core client for 30 sec - exiting 12:37:44 (4692): No heartbeat from core client for 30 sec - exiting 12:37:59 (4692): No heartbeat from core client for 30 sec - exiting 12:38:02 (4692): No heartbeat from core client for 30 sec - exiting 13:26:26 (3512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:10 (9544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:08 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:56 (10504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:14 (10504): No heartbeat from core client for 30 sec - exiting 10:07:15 (10504): No heartbeat from core client for 30 sec - exiting 10:07:16 (10504): No heartbeat from core client for 30 sec - exiting 10:07:17 (10504): No heartbeat from core client for 30 sec - exiting 10:07:18 (10504): No heartbeat from core client for 30 sec - exiting 10:07:19 (10504): No heartbeat from core client for 30 sec - exiting 10:07:20 (10504): No heartbeat from core client for 30 sec - exiting 10:07:21 (10504): No heartbeat from core client for 30 sec - exiting 10:07:22 (10504): No heartbeat from core client for 30 sec - exiting 10:49:51 (1000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:49:57 (1000): No heartbeat from core client for 30 sec - exiting 10:49:58 (1000): No heartbeat from core client for 30 sec - exiting 10:50:03 (1000): No heartbeat from core client for 30 sec - exiting 10:50:05 (1000): No heartbeat from core client for 30 sec - exiting 10:50:07 (1000): No heartbeat from core client for 30 sec - exiting 10:50:09 (1000): No heartbeat from core client for 30 sec - exiting 10:50:10 (1000): No heartbeat from core client for 30 sec - exiting 11:34:20 (7340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:25 (7340): No heartbeat from core client for 30 sec - exiting 11:34:26 (7340): No heartbeat from core client for 30 sec - exiting 11:34:27 (7340): No heartbeat from core client for 30 sec - exiting 11:34:28 (7340): No heartbeat from core client for 30 sec - exiting 11:34:29 (7340): No heartbeat from core client for 30 sec - exiting 11:34:30 (7340): No heartbeat from core client for 30 sec - exiting 11:34:31 (7340): No heartbeat from core client for 30 sec - exiting 12:18:13 (5184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:18:15 (5184): No heartbeat from core client for 30 sec - exiting 12:18:16 (5184): No heartbeat from core client for 30 sec - exiting 12:18:17 (5184): No heartbeat from core client for 30 sec - exiting 13:00:34 (3728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:42 (3728): No heartbeat from core client for 30 sec - exiting 13:00:50 (3728): No heartbeat from core client for 30 sec - exiting 13:00:57 (3728): No heartbeat from core client for 30 sec - exiting 13:00:59 (3728): No heartbeat from core client for 30 sec - exiting 13:01:03 (3728): No heartbeat from core client for 30 sec - exiting 13:01:05 (3728): No heartbeat from core client for 30 sec - exiting 13:43:22 (2188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:43:29 (2188): No heartbeat from core client for 30 sec - exiting 13:43:36 (2188): No heartbeat from core client for 30 sec - exiting 13:43:40 (2188): No heartbeat from core client for 30 sec - exiting 13:43:41 (2188): No heartbeat from core client for 30 sec - exiting 13:43:45 (2188): No heartbeat from core client for 30 sec - exiting 14:26:22 (8552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:26:32 (8552): No heartbeat from core client for 30 sec - exiting 14:26:51 (8552): No heartbeat from core client for 30 sec - exiting 14:26:58 (8552): No heartbeat from core client for 30 sec - exiting 14:27:08 (8552): No heartbeat from core client for 30 sec - exiting 15:10:03 (7536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:54:21 (9148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:54:27 (9148): No heartbeat from core client for 30 sec - exiting 15:54:28 (9148): No heartbeat from core client for 30 sec - exiting 15:54:29 (9148): No heartbeat from core client for 30 sec - exiting 15:54:33 (9148): No heartbeat from core client for 30 sec - exiting 15:54:35 (9148): No heartbeat from core client for 30 sec - exiting 15:54:36 (9148): No heartbeat from core client for 30 sec - exiting 15:54:37 (9148): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8860, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5452, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:53:55 (7036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F33AB3 read attempt to address 0x40C43447 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_n04z_1960_40_008341239/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Apr 2013 01:54:31	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	259,200	321,824	1.2416
07 Apr 2013 18:01:41	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	233,280	294,590	1.2628
07 Apr 2013 06:52:17	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	207,360	266,204	1.2838
06 Apr 2013 22:39:21	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	181,440	236,896	1.3056
06 Apr 2013 14:55:33	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	155,520	207,617	1.3350
06 Apr 2013 05:32:20	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	129,600	177,850	1.3723
05 Apr 2013 19:14:19	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	103,680	144,070	1.3896
04 Apr 2013 12:32:06	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	77,760	105,723	1.3596
03 Apr 2013 16:21:08	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	51,840	71,795	1.3849
03 Apr 2013 05:19:09	1274005	15700053	hadcm3n_n04z_1960_40_008341239_1	25,920	34,651	1.3368