Task 14069205

Name	hadcm3n_t1tr_1940_40_007751601_3
Workunit	7906710
Created	6 Feb 2012, 8:30:54 UTC
Sent	6 Feb 2012, 11:52:49 UTC
Report deadline	7 May 2012, 19:20:00 UTC
Received	9 Apr 2012, 17:36:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1043221
Run time	11 days 8 hours 55 min 39 sec
CPU time	8 days 5 hours 29 min 5 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3036, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5620, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5432, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4956, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CBUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t1trko.pje4c10 Error converting file to netcdf: dataout/t1trko.pie4c10 Error converting file to netcdf: dataout/t1trko.pfe4c10 Error converting file to netcdf: dataout/t1trka.phe4c10 Error converting file to netcdf: dataout/t1trka.pge4c10 Error converting file to netcdf: dataout/t1trka.pee4c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN p20:49:16 (5876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:49 (3012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5496, iMonCtr=1 Model crash detected, will try to restart... 12:11:56 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:57 (5244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CCPDN Monitor - Quit request from BOINC... 07:03:26 (1800): No heartbeat from core client for 30 sec - exiting 07:03:27 (1800): No heartbeat from core client for 30 sec - exiting 07:03:28 (1800): No heartbeat from core client for 30 sec - exiting 07:03:29 (1800): No heartbeat from core client for 30 sec - exiting 07:03:30 (1800): No heartbeat from core client for 30 sec - exiting 07:03:31 (1800): No heartbeat from core client for 30 sec - exiting 07:03:32 (1800): No heartbeat from core client for 30 sec - exiting 07:03:33 (1800): No heartbeat from core client for 30 sec - exiting 07:03:34 (1800): No heartbeat from core client for 30 sec - exiting 07:03:35 (1800): No heartbeat from core client for 30 sec - exiting 07:03:36 (1800): No heartbeat from core client for 30 sec - exiting 07:03:38 (1800): No heartbeat from core client for 30 sec - exiting 07:03:39 (1800): No heartbeat from core client for 30 sec - exiting 07:03:40 (1800): No heartbeat from core client for 30 sec - exiting 07:03:41 (1800): No heartbeat from core client for 30 sec - exiting 07:03:42 (1800): No heartbeat from core client for 30 sec - exiting 07:03:43 (1800): No heartbeat from core client for 30 sec - exiting 07:03:44 (1800): No heartbeat from core client for 30 sec - exiting 07:03:45 (1800): No heartbeat from core client for 30 sec - exiting 07:03:46 (1800): No heartbeat from core client for 30 sec - exiting 07:03:47 (1800): No heartbeat from core client for 30 sec - exiting 07:03:48 (1800): No heartbeat from core client for 30 sec - exiting 07:03:49 (1800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:12 (1620): No heartbeat from core client for 30 sec - exiting 13:17:13 (1620): No heartbeat from core client for 30 sec - exiting 13:17:14 (1620): No heartbeat from core client for 30 sec - exiting 13:17:15 (1620): No heartbeat from core client for 30 sec - exiting 13:17:16 (1620): No heartbeat from core client for 30 sec - exiting 13:17:17 (1620): No heartbeat from core client for 30 sec - exiting 13:17:18 (1620): No heartbeat from core client for 30 sec - exiting 13:17:19 (1620): No heartbeat from core client for 30 sec - exiting 13:17:20 (1620): No heartbeat from core client for 30 sec - exiting 13:17:21 (1620): No heartbeat from core client for 30 sec - exiting 13:17:22 (1620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:10:34 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:24:29 (4884): No heartbeat from core client for 30 sec - exiting 09:24:30 (4884): No heartbeat from core client for 30 sec - exiting 09:24:31 (4884): No heartbeat from core client for 30 sec - exiting 09:24:32 (4884): No heartbeat from core client for 30 sec - exiting 09:24:33 (4884): No heartbeat from core client for 30 sec - exiting 09:24:34 (4884): No heartbeat from core client for 30 sec - exiting 09:24:35 (4884): No heartbeat from core client for 30 sec - exiting 09:24:36 (4884): No heartbeat from core client for 30 sec - exiting 09:24:37 (4884): No heartbeat from core client for 30 sec - exiting 09:24:38 (4884): No heartbeat from core client for 30 sec - exiting 09:24:39 (4884): No heartbeat from core client for 30 sec - exiting 09:24:40 (4884): No heartbeat from core client for 30 sec - exiting 09:24:41 (4884): No heartbeat from core client for 30 sec - exiting 09:24:42 (4884): No heartbeat from core client for 30 sec - exiting 09:24:43 (4884): No heartbeat from core client for 30 sec - exiting 09:24:44 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:49:33 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:17 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... 08:14:51 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:23 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4440, iMonCtr=1 Model crash detected, will try to restart... 20:19:23 (1532): No heartbeat from core client for 30 sec - exiting 20:19:24 (1532): No heartbeat from core client for 30 sec - exiting 20:19:25 (1532): No heartbeat from core client for 30 sec - exiting 20:19:26 (1532): No heartbeat from core client for 30 sec - exiting 20:19:27 (1532): No heartbeat from core client for 30 sec - exiting 20:19:28 (1532): No heartbeat from core client for 30 sec - exiting 20:19:29 (1532): No heartbeat from core client for 30 sec - exiting 20:19:30 (1532): No heartbeat from core client for 30 sec - exiting 20:19:31 (1532): No heartbeat from core client for 30 sec - exiting 20:19:32 (1532): No heartbeat from core client for 30 sec - exiting 20:19:33 (1532): No heartbeat from core client for 30 sec - exiting 20:19:34 (1532): No heartbeat from core client for 30 sec - exiting 20:19:35 (1532): No heartbeat from core client for 30 sec - exiting 20:19:36 (1532): No heartbeat from core client for 30 sec - exiting 20:19:37 (1532): No heartbeat from core client for 30 sec - exiting 20:19:38 (1532): No heartbeat from core client for 30 sec - exiting 20:19:39 (1532): No heartbeat from core client for 30 sec - exiting 20:19:40 (1532): No heartbeat from core client for 30 sec - exiting 20:19:41 (1532): No heartbeat from core client for 30 sec - exiting 20:19:42 (1532): No heartbeat from core client for 30 sec - exiting 20:19:43 (1532): No heartbeat from core client for 30 sec - exiting 20:19:44 (1532): No heartbeat from core client for 30 sec - exiting 20:19:45 (1532): No heartbeat from core client for 30 sec - exiting 20:19:46 (1532): No heartbeat from core client for 30 sec - exiting 20:19:47 (1532): No heartbeat from core client for 30 sec - exiting 20:19:48 (1532): No heartbeat from core client for 30 sec - exiting 20:19:49 (1532): No heartbeat from core client for 30 sec - exiting 20:19:50 (1532): No heartbeat from core client for 30 sec - exiting 20:19:51 (1532): No heartbeat from core client for 30 sec - exiting 20:19:52 (1532): No heartbeat from core client for 30 sec - exiting 20:19:53 (1532): No heartbeat from core client for 30 sec - exiting 20:19:54 (1532): No heartbeat from core client for 30 sec - exiting 20:19:55 (1532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6040, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4240, iMonCtr=1 Model crash detected, will try to restart... 09:50:06 (4252): No heartbeat from core client for 30 sec - exiting 09:50:08 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:09 (4252): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5956, iMonCtr=1 Model crash detected, will try to restart... 15:20:54 (5596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5608, iMonCtr=1 Model crash detected, will try to restart... 17:29:59 (5980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:51:06 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:03:48 (5524): No heartbeat from core client for 30 sec - exiting 08:03:49 (5524): No heartbeat from core client for 30 sec - exiting 08:03:50 (5524): No heartbeat from core client for 30 sec - exiting 08:03:51 (5524): No heartbeat from core client for 30 sec - exiting 08:03:52 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:54 (5524): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4708, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 12:00:11 (720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:11 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:17 (5732): No heartbeat from core client for 30 sec - exiting 08:23:18 (5732): No heartbeat from core client for 30 sec - exiting 08:23:19 (5732): No heartbeat from core client for 30 sec - exiting 08:23:20 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5020, iMonCtr=1 Model crash detected, will try to restart... 08:02:17 (5980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4840, iMonCtr=1 Model crash detected, will try to restart... 14:44:51 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Apr 2012 18:23:29	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	259,200	710,915	2.7427
04 Apr 2012 11:30:13	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	233,280	645,717	2.7680
30 Mar 2012 10:53:42	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	207,360	578,118	2.7880
26 Mar 2012 09:52:29	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	181,440	510,411	2.8131
20 Mar 2012 09:39:24	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	155,520	442,535	2.8455
12 Mar 2012 09:27:07	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	129,600	371,788	2.8687
04 Mar 2012 20:27:12	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	103,680	295,717	2.8522
27 Feb 2012 08:10:57	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	77,760	223,945	2.8800
22 Feb 2012 16:12:09	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	51,840	146,766	2.8311
13 Feb 2012 20:14:33	1043221	14069205	hadcm3n_t1tr_1940_40_007751601_3	25,920	75,094	2.8971