Task 13464211

Name	hadcm3n_o1zg_1940_40_007442993_2
Workunit	7640496
Created	6 Oct 2011, 14:48:59 UTC
Sent	6 Oct 2011, 14:49:17 UTC
Report deadline	5 Jan 2012, 22:16:28 UTC
Received	30 Nov 2011, 20:03:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	953777
Run time	15 days 3 hours 13 min 6 sec
CPU time	14 days 1 hours 29 min 46 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.43</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 18:39:59 (5596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:00 (5596): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3972, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6732, iMonCtr=1 Model crash detected, will try to restart... 11:21:03 (2820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:10:20 (4520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:10:22 (4520): No heartbeat from core client for 30 sec - exiting 15:10:23 (4520): No heartbeat from core client for 30 sec - exiting 15:10:24 (4520): No heartbeat from core client for 30 sec - exiting 15:10:25 (4520): No heartbeat from core client for 30 sec - exiting 15:10:26 (4520): No heartbeat from core client for 30 sec - exiting 15:10:27 (4520): No heartbeat from core client for 30 sec - exiting 15:11:15 (6904): Can't set up shared mem: -1. Will run in standalone mode. 15:11:56 (7736): Can't set up shared mem: -1. Will run in standalone mode. 18:19:36 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:25 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:25:17 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:05 (4620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:06 (4620): No heartbeat from core client for 30 sec - exiting 13:13:07 (4620): No heartbeat from core client for 30 sec - exiting 13:13:08 (4620): No heartbeat from core client for 30 sec - exiting 13:14:58 (2704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:12 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:50 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:53 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:52 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:38:43 (7856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:07 (6168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:02 (6388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:32 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:31 (7872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:49:23 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 16:22:21 (7440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 17:09:16 (7936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Nov 2011 11:51:32	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	518,400	1,214,981	2.3437
24 Nov 2011 15:53:30	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	492,480	1,145,236	2.3254
21 Nov 2011 16:10:26	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	466,560	1,075,314	2.3048
15 Nov 2011 17:25:28	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	440,640	1,005,195	2.2812
15 Nov 2011 17:25:28	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	414,720	942,327	2.2722
15 Nov 2011 17:25:27	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	388,800	870,603	2.2392
05 Nov 2011 14:57:48	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	362,880	798,866	2.2015
31 Oct 2011 15:59:32	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	336,960	717,735	2.1300
31 Oct 2011 13:12:58	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	311,040	651,420	2.0943
31 Oct 2011 13:12:58	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	285,120	604,159	2.1190
31 Oct 2011 13:12:58	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	259,200	556,495	2.1470
31 Oct 2011 13:12:58	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	233,280	508,501	2.1798
31 Oct 2011 13:12:58	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	207,360	460,753	2.2220
14 Oct 2011 20:19:34	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	181,440	406,515	2.2405
12 Oct 2011 11:22:50	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	155,520	356,100	2.2897
10 Oct 2011 21:59:03	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	129,600	294,986	2.2761
09 Oct 2011 18:19:14	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	103,680	232,320	2.2407
09 Oct 2011 00:29:15	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	77,760	167,874	2.1589
08 Oct 2011 08:17:53	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	51,840	103,334	1.9933
07 Oct 2011 16:00:25	953777	13464211	hadcm3n_o1zg_1940_40_007442993_2	25,920	47,190	1.8206