Task 15660689

Name	hadcm3n_zm1x_1920_40_008280054_2
Workunit	8431189
Created	12 Mar 2013, 20:58:34 UTC
Sent	12 Mar 2013, 21:09:17 UTC
Report deadline	12 Jun 2013, 4:36:28 UTC
Received	8 Apr 2013, 7:21:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1209561
Run time	21 days 7 hours 45 min 33 sec
CPU time	10 days 17 hours 52 min 14 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6180, iMonCtr=1 Model crash detected, will try to restart... 03:35:50 (5016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5348, iMonCtr=1 Model crash detected, will try to restart... 09:12:05 (1236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:06 (1236): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4004, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6628, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1456, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4008, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 13:22:43 (6744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6444, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... 14:46:47 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6364, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:16:03 (3940): No heartbeat from core client for 30 sec - exiting 16:16:05 (3940): No heartbeat from core client for 30 sec - exiting 16:16:06 (3940): No heartbeat from core client for 30 sec - exiting 16:16:07 (3940): No heartbeat from core client for 30 sec - exiting 16:16:08 (3940): No heartbeat from core client for 30 sec - exiting 16:16:09 (3940): No heartbeat from core client for 30 sec - exiting 16:16:10 (3940): No heartbeat from core client for 30 sec - exiting 16:16:12 (3940): No heartbeat from core client for 30 sec - exiting 16:16:13 (3940): No heartbeat from core client for 30 sec - exiting 16:16:14 (3940): No heartbeat from core client for 30 sec - exiting 16:16:15 (3940): No heartbeat from core client for 30 sec - exiting 16:16:16 (3940): No heartbeat from core client for 30 sec - exiting 16:16:17 (3940): No heartbeat from core client for 30 sec - exiting 16:16:18 (3940): No heartbeat from core client for 30 sec - exiting 16:16:19 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4776, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:51:44 (4524): No heartbeat from core client for 30 sec - exiting 13:51:45 (4524): No heartbeat from core client for 30 sec - exiting 13:51:46 (4524): No heartbeat from core client for 30 sec - exiting 13:51:47 (4524): No heartbeat from core client for 30 sec - exiting 13:51:48 (4524): No heartbeat from core client for 30 sec - exiting 13:51:49 (4524): No heartbeat from core client for 30 sec - exiting 13:51:50 (4524): No heartbeat from core client for 30 sec - exiting 13:51:51 (4524): No heartbeat from core client for 30 sec - exiting 13:51:52 (4524): No heartbeat from core client for 30 sec - exiting 13:51:53 (4524): No heartbeat from core client for 30 sec - exiting 13:51:54 (4524): No heartbeat from core client for 30 sec - exiting 13:51:55 (4524): No heartbeat from core client for 30 sec - exiting 13:51:56 (4524): No heartbeat from core client for 30 sec - exiting 13:51:57 (4524): No heartbeat from core client for 30 sec - exiting 13:51:58 (4524): No heartbeat from core client for 30 sec - exiting 13:51:59 (4524): No heartbeat from core client for 30 sec - exiting 13:52:00 (4524): No heartbeat from core client for 30 sec - exiting 13:52:02 (4524): No heartbeat from core client for 30 sec - exiting 13:52:03 (4524): No heartbeat from core client for 30 sec - exiting 13:52:04 (4524): No heartbeat from core client for 30 sec - exiting 13:52:05 (4524): No heartbeat from core client for 30 sec - exiting 13:52:06 (4524): No heartbeat from core client for 30 sec - exiting 13:52:07 (4524): No heartbeat from core client for 30 sec - exiting 13:52:09 (4524): No heartbeat from core client for 30 sec - exiting 13:52:10 (4524): No heartbeat from core client for 30 sec - exiting 13:52:11 (4524): No heartbeat from core client for 30 sec - exiting 13:52:13 (4524): No heartbeat from core client for 30 sec - exiting 13:52:14 (4524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:07:01 (1180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:03 (1180): No heartbeat from core client for 30 sec - exiting 20:07:04 (1180): No heartbeat from core client for 30 sec - exiting 20:07:05 (1180): No heartbeat from core client for 30 sec - exiting 20:08:08 (2484): No heartbeat from core client for 30 sec - exiting 20:08:09 (2484): No heartbeat from core client for 30 sec - exiting 20:08:10 (2484): No heartbeat from core client for 30 sec - exiting 20:08:11 (2484): No heartbeat from core client for 30 sec - exiting 20:08:12 (2484): No heartbeat from core client for 30 sec - exiting 20:08:13 (2484): No heartbeat from core client for 30 sec - exiting 20:08:14 (2484): No heartbeat from core client for 30 sec - exiting 20:08:15 (2484): No heartbeat from core client for 30 sec - exiting 20:08:16 (2484): No heartbeat from core client for 30 sec - exiting 20:08:17 (2484): No heartbeat from core client for 30 sec - exiting 20:08:18 (2484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774E3AB3 read attempt to address 0x40AFB156 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zm1x_1920_40_008280054/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Apr 2013 13:09:20	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	518,400	928,311	1.7907
06 Apr 2013 07:38:12	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	492,480	882,420	1.7918
05 Apr 2013 05:48:25	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	466,560	835,913	1.7917
04 Apr 2013 00:12:18	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	440,640	789,246	1.7911
03 Apr 2013 00:57:27	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	414,720	743,529	1.7928
01 Apr 2013 20:07:46	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	388,800	695,622	1.7892
30 Mar 2013 16:05:44	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	362,880	649,886	1.7909
29 Mar 2013 08:45:54	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	336,960	603,962	1.7924
27 Mar 2013 17:31:30	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	311,040	557,246	1.7916
26 Mar 2013 11:22:06	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	285,120	511,555	1.7942
25 Mar 2013 03:15:53	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	259,200	465,899	1.7974
24 Mar 2013 06:01:18	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	233,280	421,424	1.8065
23 Mar 2013 02:45:55	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	207,360	372,373	1.7958
20 Mar 2013 12:22:56	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	181,440	325,180	1.7922
19 Mar 2013 12:14:56	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	155,520	278,674	1.7919
18 Mar 2013 07:08:15	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	129,600	233,437	1.8012
17 Mar 2013 08:28:51	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	103,680	187,551	1.8089
16 Mar 2013 06:22:25	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	77,760	140,710	1.8095
15 Mar 2013 02:01:36	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	51,840	93,490	1.8034
14 Mar 2013 02:58:08	1209561	15660689	hadcm3n_zm1x_1920_40_008280054_2	25,920	47,006	1.8135