Task 12742066

Name	hadcm3n_o3xs_1900_40_007200435_0
Workunit	7398715
Created	28 Mar 2011, 14:08:33 UTC
Sent	30 Mar 2011, 22:07:40 UTC
Report deadline	30 Jun 2011, 5:34:51 UTC
Received	8 May 2011, 22:49:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1107482
Run time	4 days 19 hours 47 min 25 sec
CPU time	4 days 17 hours 29 min 36 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not ru08:39:51 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:43 (3420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:26:08 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:41:30 (4316): No heartbeat from core client for 30 sec - exiting 10:41:31 (4316): No heartbeat from core client for 30 sec - exiting 10:41:32 (4316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:02:53 (3224): No heartbeat from core client for 30 sec - exiting 14:02:54 (3224): No heartbeat from core client for 30 sec - exiting 14:02:55 (3224): No heartbeat from core client for 30 sec - exiting 14:02:56 (3224): No heartbeat from core client for 30 sec - exiting 14:02:57 (3224): No heartbeat from core client for 30 sec - exiting 14:02:58 (3224): No heartbeat from core client for 30 sec - exiting 14:02:59 (3224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:44 (1696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:14 (4100): No heartbeat from core client for 30 sec - exiting 18:43:16 (4100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:10:30 (2708): No heartbeat from core client for 30 sec - exiting 13:10:31 (2708): No heartbeat from core client for 30 sec - exiting 13:10:32 (2708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:04:31 (3116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:41 (3716): No heartbeat from core client for 30 sec - exiting 19:20:42 (3716): No heartbeat from core client for 30 sec - exiting 19:20:43 (3716): No heartbeat from core client for 30 sec - exiting 19:20:44 (3716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:15 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:50:23 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:40:02 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:57:48 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:48:02 (4420): No heartbeat from core client for 30 sec - exiting 09:48:03 (4420): No heartbeat from core client for 30 sec - exiting 09:48:04 (4420): No heartbeat from core client for 30 sec - exiting 09:48:05 (4420): No heartbeat from core client for 30 sec - exiting 09:48:06 (4420): No heartbeat from core client for 30 sec - exiting 09:48:07 (4420): No heartbeat from core client for 30 sec - exiting 09:48:08 (4420): No heartbeat from core client for 30 sec - exiting 09:48:09 (4420): No heartbeat from core client for 30 sec - exiting 09:48:10 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3316, iMonCtr=1 Model crash detected, will try to restart... 15:31:26 (4708): No heartbeat from core client for 30 sec - exiting 15:31:27 (4708): No heartbeat from core client for 30 sec - exiting 15:31:28 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:45:43 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4440, iMonCtr=1 Model crash detected, will try to restart... 10:30:09 (1152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4564, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:14:36 (3980): No heartbeat from core client for 30 sec - exiting 11:14:37 (3980): No heartbeat from core client for 30 sec - exiting 11:14:38 (3980): No heartbeat from core client for 30 sec - exiting 11:14:39 (3980): No heartbeat from core client for 30 sec - exiting 11:14:40 (3980): No heartbeat from core client for 30 sec - exiting 11:14:42 (3980): No heartbeat from core client for 30 sec - exiting 11:14:43 (3980): No heartbeat from core client for 30 sec - exiting 11:14:44 (3980): No heartbeat from core client for 30 sec - exiting 11:14:45 (3980): No heartbeat from core client for 30 sec - exiting 11:14:46 (3980): No heartbeat from core client for 30 sec - exiting 11:14:47 (3980): No heartbeat from core client for 30 sec - exiting 11:14:48 (3980): No heartbeat from core client for 30 sec - exiting 11:14:49 (3980): No heartbeat from core client for 30 sec - exiting 11:14:50 (3980): No heartbeat from core client for 30 sec - exiting 11:14:51 (3980): No heartbeat from core client for 30 sec - exiting 11:14:53 (3980): No heartbeat from core client for 30 sec - exiting 11:14:54 (3980): No heartbeat from core client for 30 sec - exiting 11:14:55 (3980): No heartbeat from core client for 30 sec - exiting 11:14:56 (3980): No heartbeat from core client for 30 sec - exiting 11:14:57 (3980): No heartbeat from core client for 30 sec - exiting 11:14:58 (3980): No heartbeat from core client for 30 sec - exiting 11:14:59 (3980): No heartbeat from core client for 30 sec - exiting 11:15:00 (3980): No heartbeat from core client for 30 sec - exiting 11:15:01 (3980): No heartbeat from core client for 30 sec - exiting 11:15:02 (3980): No heartbeat from core client for 30 sec - exiting 11:15:03 (3980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:41 (4220): No heartbeat from core client for 30 sec - exiting 11:20:42 (4220): No heartbeat from core client for 30 sec - exiting 11:20:43 (4220): No heartbeat from core client for 30 sec - exiting 11:20:44 (4220): No heartbeat from core client for 30 sec - exiting 11:20:45 (4220): No heartbeat from core client for 30 sec - exiting 11:20:46 (4220): No heartbeat from core client for 30 sec - exiting 11:20:47 (4220): No heartbeat from core client for 30 sec - exiting 11:20:48 (4220): No heartbeat from core client for 30 sec - exiting 11:20:49 (4220): No heartbeat from core client for 30 sec - exiting 11:20:50 (4220): No heartbeat from core client for 30 sec - exiting 11:20:51 (4220): No heartbeat from core client for 30 sec - exiting 11:20:52 (4220): No heartbeat from core client for 30 sec - exiting 11:20:53 (4220): No heartbeat from core client for 30 sec - exiting 11:20:54 (4220): No heartbeat from core client for 30 sec - exiting 11:20:55 (4220): No heartbeat from core client for 30 sec - exiting 11:20:56 (4220): No heartbeat from core client for 30 sec - exiting 11:20:57 (4220): No heartbeat from core client for 30 sec - exiting 11:20:58 (4220): No heartbeat from core client for 30 sec - exiting 11:20:59 (4220): No heartbeat from core client for 30 sec - exiting 11:21:00 (4220): No heartbeat from core client for 30 sec - exiting 11:21:01 (4220): No heartbeat from core client for 30 sec - exiting 11:21:02 (4220): No heartbeat from core client for 30 sec - exiting 11:21:03 (4220): No heartbeat from core client for 30 sec - exiting 11:21:04 (4220): No heartbeat from core client for 30 sec - exiting 11:21:05 (4220): No heartbeat from core client for 30 sec - exiting 11:21:06 (4220): No heartbeat from core client for 30 sec - exiting 11:21:07 (4220): No heartbeat from core client for 30 sec - exiting 11:21:08 (4220): No heartbeat from core client for 30 sec - exiting 11:21:09 (4220): No heartbeat from core client for 30 sec - exiting 11:21:10 (4220): No heartbeat from core client for 30 sec - exiting 11:21:11 (4220): No heartbeat from core client for 30 sec - exiting 11:21:12 (4220): No heartbeat from core client for 30 sec - exiting 11:21:13 (4220): No heartbeat from core client for 30 sec - exiting 11:21:14 (4220): No heartbeat from core client for 30 sec - exiting 11:21:15 (4220): No heartbeat from core client for 30 sec - exiting 11:21:16 (4220): No heartbeat from core client for 30 sec - exiting 11:21:17 (4220): No heartbeat from core client for 30 sec - exiting 11:21:18 (4220): No heartbeat from core client for 30 sec - exiting 11:21:19 (4220): No heartbeat from core client for 30 sec - exiting 11:21:20 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3888, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5576, iMonCtr=1 Model crash detected, will try to restart... 12:33:21 (3388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... 22:42:03 (5044): No heartbeat from core client for 30 sec - exiting 22:42:04 (5044): No heartbeat from core client for 30 sec - exiting 22:42:05 (5044): No heartbeat from core client for 30 sec - exiting 22:42:07 (5044): No heartbeat from core client for 30 sec - exiting 22:42:08 (5044): No heartbeat from core client for 30 sec - exiting 22:42:09 (5044): No heartbeat from core client for 30 sec - exiting 22:42:10 (5044): No heartbeat from core client for 30 sec - exiting 22:42:11 (5044): No heartbeat from core client for 30 sec - exiting 22:42:12 (5044): No heartbeat from core client for 30 sec - exiting 22:42:13 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:37:38 (3484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5656, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3624, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1996, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1124, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4908, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1480, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4616, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2604, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778F3A93 read attempt to address 0x404DEB6C Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778F3A93 read attempt to address 0x404DEB6C Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o3xs_1900_40_007200435/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 May 2011 18:18:56	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	259,200	392,489	1.5142
03 May 2011 09:14:36	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	233,280	336,926	1.4443
29 Apr 2011 07:57:01	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	207,360	299,444	1.4441
26 Apr 2011 23:43:31	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	181,440	265,583	1.4638
25 Apr 2011 15:39:02	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	155,520	232,758	1.4966
24 Apr 2011 13:31:09	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	129,600	197,565	1.5244
20 Apr 2011 17:48:49	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	103,680	158,831	1.5319
07 Apr 2011 16:01:25	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	77,760	120,255	1.5465
03 Apr 2011 15:00:20	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	51,840	81,203	1.5664
01 Apr 2011 15:33:53	1107482	12742066	hadcm3n_o3xs_1900_40_007200435_0	25,920	40,675	1.5693