Task 15558406

Name	hadcm3n_o4rz_2060_40_008203346_4
Workunit	8358470
Created	26 Jan 2013, 14:25:45 UTC
Sent	26 Jan 2013, 14:25:54 UTC
Report deadline	27 Apr 2013, 21:53:05 UTC
Received	24 Mar 2013, 10:13:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1114363
Run time	15 days 13 hours 52 min 32 sec
CPU time	14 days 15 hours 58 min 31 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.43 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:53:23 (5772): No heartbeat from core client for 30 sec - exiting 17:53:24 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3580, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5056, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:41:56 (3396): No heartbeat from core client for 30 sec - exiting 14:41:57 (3396): No heartbeat from core client for 30 sec - exiting 14:41:58 (3396): No heartbeat from core client for 30 sec - exiting 14:41:59 (3396): No heartbeat from core client for 30 sec - exiting 14:42:00 (3396): No heartbeat from core client for 30 sec - exiting 14:42:01 (3396): No heartbeat from core client for 30 sec - exiting 14:42:02 (3396): No heartbeat from core client for 30 sec - exiting 14:42:03 (3396): No heartbeat from core client for 30 sec - exiting 14:42:04 (3396): No heartbeat from core client for 30 sec - exiting 14:42:05 (3396): No heartbeat from core client for 30 sec - exiting 14:42:06 (3396): No heartbeat from core client for 30 sec - exiting 14:42:08 (3396): No heartbeat from core client for 30 sec - exiting 14:42:09 (3396): No heartbeat from core client for 30 sec - exiting 14:42:10 (3396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:26 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5220, iMonCtr=1 Model crash detected, will try to restart... 08:51:49 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:53:46 (4688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:47 (4688): No heartbeat from core client for 30 sec - exiting 02:40:13 (4892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:50:08 (4420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1116, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5972, iMonCtr=1 Model crash detected, will try to restart... 14:26:29 (5892): No heartbeat from core client for 30 sec - exiting 14:26:30 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=1 Model crash detected, will try to restart... 09:51:49 (5548): No heartbeat from core client for 30 sec - exiting 09:51:50 (5548): No heartbeat from core client for 30 sec - exiting 09:51:52 (5548): No heartbeat from core client for 30 sec - exiting 09:51:53 (5548): No heartbeat from core client for 30 sec - exiting 09:51:54 (5548): No heartbeat from core client for 30 sec - exiting 09:51:55 (5548): No heartbeat from core client for 30 sec - exiting 09:51:56 (5548): No heartbeat from core client for 30 sec - exiting 09:51:57 (5548): No heartbeat from core client for 30 sec - exiting 09:51:58 (5548): No heartbeat from core client for 30 sec - exiting 09:51:59 (5548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77233AB3 read attempt to address 0x40977FEC Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o4rz_2060_40_008203346/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Mar 2013 02:09:53	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	518,400	1,267,104	2.4443
18 Mar 2013 13:04:59	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	492,480	1,193,733	2.4239
17 Mar 2013 16:30:46	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	466,560	1,121,629	2.4040
10 Mar 2013 03:40:23	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	440,640	1,050,656	2.3844
03 Mar 2013 16:00:43	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	414,720	984,155	2.3731
02 Mar 2013 20:40:55	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	388,800	917,582	2.3600
02 Mar 2013 02:35:33	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	362,880	852,509	2.3493
25 Feb 2013 19:38:54	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	336,960	780,615	2.3166
24 Feb 2013 13:30:24	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	311,040	715,458	2.3002
23 Feb 2013 20:08:17	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	285,120	655,409	2.2987
17 Feb 2013 11:37:31	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	259,200	590,043	2.2764
16 Feb 2013 19:00:46	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	233,280	530,907	2.2758
07 Feb 2013 11:18:49	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	207,360	467,892	2.2564
06 Feb 2013 11:43:42	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	181,440	405,156	2.2330
05 Feb 2013 13:20:18	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	155,520	345,376	2.2208
04 Feb 2013 18:11:47	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	129,600	287,417	2.2177
04 Feb 2013 02:26:19	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	103,680	231,297	2.2309
01 Feb 2013 07:41:37	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	77,760	175,129	2.2522
31 Jan 2013 17:21:14	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	51,840	121,184	2.3377
27 Jan 2013 08:40:59	1114363	15558406	hadcm3n_o4rz_2060_40_008203346_4	25,920	60,531	2.3353