Task 11432656

Name	famous_u2pz_1399_200_006636370_1
Workunit	6839742
Created	10 Jun 2010, 11:31:17 UTC
Sent	19 Jul 2010, 18:48:32 UTC
Report deadline	19 Oct 2010, 2:15:43 UTC
Received	26 Jul 2010, 1:59:44 UTC
Server state	Over
Outcome	Computation error
Client state	Done
Exit status	22 (0x00000016) Unknown error code
Computer ID	986959
Run time	4 days 11 hours 55 min 46 sec
CPU time	4 days 11 hours 55 min 46 sec
Validate state	Invalid
Credit	3,366.18
Device peak FLOPS	2.56 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>5.8.15</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 05:37:17 (5124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:37:18 (5124): No heartbeat from core client for 30 sec - exiting 05:37:19 (5124): No heartbeat from core client for 30 sec - exiting 05:37:21 (5124): No heartbeat from core client for 30 sec - exiting 05:37:26 (5124): No heartbeat from core client for 30 sec - exiting 05:37:27 (5124): No heartbeat from core client for 30 sec - exiting 05:37:28 (5124): No heartbeat from core client for 30 sec - exiting 05:37:29 (5124): No heartbeat from core client for 30 sec - exiting 06:18:15 (5588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:16 (5588): No heartbeat from core client for 30 sec - exiting 06:18:18 (5588): No heartbeat from core client for 30 sec - exiting 06:18:19 (5588): No heartbeat from core client for 30 sec - exiting 06:18:20 (5588): No heartbeat from core client for 30 sec - exiting 06:18:21 (5588): No heartbeat from core client for 30 sec - exiting 06:18:22 (5588): No heartbeat from core client for 30 sec - exiting 06:18:23 (5588): No heartbeat from core client for 30 sec - exiting 06:18:24 (5588): No heartbeat from core client for 30 sec - exiting 06:26:49 (7084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:01 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:11 (7860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:27 (7972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:19:22 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:19:27 (4252): No heartbeat from core client for 30 sec - exiting 07:19:28 (4252): No heartbeat from core client for 30 sec - exiting 07:19:29 (4252): No heartbeat from core client for 30 sec - exiting 07:19:31 (4252): No heartbeat from core client for 30 sec - exiting 07:19:32 (4252): No heartbeat from core client for 30 sec - exiting 07:27:36 (6964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:07 (6964): No heartbeat from core client for 30 sec - exiting 07:28:10 (6964): No heartbeat from core client for 30 sec - exiting 07:28:11 (6964): No heartbeat from core client for 30 sec - exiting 07:28:12 (6964): No heartbeat from core client for 30 sec - exiting 07:28:14 (6964): No heartbeat from core client for 30 sec - exiting 07:28:15 (6964): No heartbeat from core client for 30 sec - exiting 07:28:16 (6964): No heartbeat from core client for 30 sec - exiting 07:28:19 (6964): No heartbeat from core client for 30 sec - exiting 07:34:36 (7328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:30 (7028): No heartbeat from core client for 30 sec - exiting 07:54:32 (7028): No heartbeat from core client for 30 sec - exiting 07:54:34 (7028): No heartbeat from core client for 30 sec - exiting 07:54:36 (7028): No heartbeat from core client for 30 sec - exiting 07:54:37 (7028): No heartbeat from core client for 30 sec - exiting 07:54:39 (7028): No heartbeat from core client for 30 sec - exiting 07:54:40 (7028): No heartbeat from core client for 30 sec - exiting 07:54:41 (7028): No heartbeat from core client for 30 sec - exiting 07:54:42 (7028): No heartbeat from core client for 30 sec - exiting 07:54:44 (7028): No heartbeat from core client for 30 sec - exiting 07:54:45 (7028): No heartbeat from core client for 30 sec - exiting 07:54:46 (7028): No heartbeat from core client for 30 sec - exiting 07:54:48 (7028): No heartbeat from core client for 30 sec - exiting 07:54:49 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... 08:04:30 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:04:31 (3424): No heartbeat from core client for 30 sec - exiting 08:04:33 (3424): No heartbeat from core client for 30 sec - exiting 08:10:51 (5064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:22:11 (6836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:42 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:46 (4264): No heartbeat from core client for 30 sec - exiting 08:29:47 (4264): No heartbeat from core client for 30 sec - exiting 08:29:48 (4264): No heartbeat from core client for 30 sec - exiting 08:29:49 (4264): No heartbeat from core client for 30 sec - exiting 08:29:50 (4264): No heartbeat from core client for 30 sec - exiting 08:29:51 (4264): No heartbeat from core client for 30 sec - exiting 08:29:52 (4264): No heartbeat from core client for 30 sec - exiting 08:29:53 (4264): No heartbeat from core client for 30 sec - exiting 08:29:54 (4264): No heartbeat from core client for 30 sec - exiting 08:29:55 (4264): No heartbeat from core client for 30 sec - exiting 08:33:07 (7940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:38 (7372): No heartbeat from core client for 30 sec - exiting 08:40:46 (7372): No heartbeat from core client for 30 sec - exiting 08:40:47 (7372): No heartbeat from core client for 30 sec - exiting 08:40:48 (7372): No heartbeat from core client for 30 sec - exiting 08:40:49 (7372): No heartbeat from core client for 30 sec - exiting 08:40:50 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:39 (2432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:45 (2432): No heartbeat from core client for 30 sec - exiting 08:45:46 (2432): No heartbeat from core client for 30 sec - exiting 08:50:34 (312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:18:59 (6192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:02 (6192): No heartbeat from core client for 30 sec - exiting 04:19:03 (6192): No heartbeat from core client for 30 sec - exiting 04:19:04 (6192): No heartbeat from core client for 30 sec - exiting 04:19:05 (6192): No heartbeat from core client for 30 sec - exiting 04:19:06 (6192): No heartbeat from core client for 30 sec - exiting 04:19:07 (6192): No heartbeat from core client for 30 sec - exiting 04:19:10 (6192): No heartbeat from core client for 30 sec - exiting 04:19:11 (6192): No heartbeat from core client for 30 sec - exiting 04:37:29 (2340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:46:50 (756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:46:56 (756): No heartbeat from core client for 30 sec - exiting 04:46:57 (756): No heartbeat from core client for 30 sec - exiting 04:46:58 (756): No heartbeat from core client for 30 sec - exiting 04:46:59 (756): No heartbeat from core client for 30 sec - exiting 04:47:00 (756): No heartbeat from core client for 30 sec - exiting 04:47:02 (756): No heartbeat from core client for 30 sec - exiting 04:47:03 (756): No heartbeat from core client for 30 sec - exiting 04:47:04 (756): No heartbeat from core client for 30 sec - exiting 04:47:05 (756): No heartbeat from core client for 30 sec - exiting 04:47:07 (756): No heartbeat from core client for 30 sec - exiting 04:49:03 (7780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:59:06 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:59:10 (6992): No heartbeat from core client for 30 sec - exiting 04:59:11 (6992): No heartbeat from core client for 30 sec - exiting 04:59:12 (6992): No heartbeat from core client for 30 sec - exiting 04:59:13 (6992): No heartbeat from core client for 30 sec - exiting 04:59:14 (6992): No heartbeat from core client for 30 sec - exiting 04:59:15 (6992): No heartbeat from core client for 30 sec - exiting 04:59:16 (6992): No heartbeat from core client for 30 sec - exiting 05:33:05 (6236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:50:30 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1944, iMonCtr=1 Model crash detected, will try to restart... 05:54:24 (1944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:25 (1944): No heartbeat from core client for 30 sec - exiting 05:54:26 (1944): No heartbeat from core client for 30 sec - exiting 06:04:28 (2396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:14:40 (7736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:54 (1204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:20 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:24 (5400): No heartbeat from core client for 30 sec - exiting 06:31:25 (5400): No heartbeat from core client for 30 sec - exiting 06:31:26 (5400): No heartbeat from core client for 30 sec - exiting 06:31:27 (5400): No heartbeat from core client for 30 sec - exiting 06:31:28 (5400): No heartbeat from core client for 30 sec - exiting 06:31:29 (5400): No heartbeat from core client for 30 sec - exiting 06:31:30 (5400): No heartbeat from core client for 30 sec - exiting 06:31:31 (5400): No heartbeat from core client for 30 sec - exiting 06:31:32 (5400): No heartbeat from core client for 30 sec - exiting 06:31:33 (5400): No heartbeat from core client for 30 sec - exiting 06:48:57 (7260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:54 (3708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:56 (6280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:11 (8056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:14 (8056): No heartbeat from core client for 30 sec - exiting 07:06:15 (8056): No heartbeat from core client for 30 sec - exiting 07:06:16 (8056): No heartbeat from core client for 30 sec - exiting 07:06:18 (8056): No heartbeat from core client for 30 sec - exiting 07:06:22 (8056): No heartbeat from core client for 30 sec - exiting 07:06:23 (8056): No heartbeat from core client for 30 sec - exiting 07:06:24 (8056): No heartbeat from core client for 30 sec - exiting 07:06:25 (8056): No heartbeat from core client for 30 sec - exiting 07:06:26 (8056): No heartbeat from core client for 30 sec - exiting 07:06:27 (8056): No heartbeat from core client for 30 sec - exiting 07:23:18 (7604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:23:21 (7604): No heartbeat from core client for 30 sec - exiting 07:23:22 (7604): No heartbeat from core client for 30 sec - exiting 07:23:23 (7604): No heartbeat from core client for 30 sec - exiting 07:23:24 (7604): No heartbeat from core client for 30 sec - exiting Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Sorry, too many model crashes! :-( 07:28:27 (684): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	1,020,266	388,408	0.3807
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	1,010,906	384,632	0.3805
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	1,001,546	380,743	0.3802
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	992,186	376,853	0.3798
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	982,826	373,343	0.3799
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	973,466	369,932	0.3800
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	964,106	366,430	0.3801
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	954,746	362,967	0.3802
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	945,386	359,525	0.3803
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	936,026	356,102	0.3804
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	926,666	352,666	0.3806
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	917,306	349,232	0.3807
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	907,946	345,807	0.3809
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	898,586	342,340	0.3810
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	889,226	338,857	0.3811
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	879,866	335,428	0.3812
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	870,506	332,006	0.3814
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	861,146	328,589	0.3816
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	851,786	325,167	0.3817
26 Jul 2010 02:04:35	986959	11432656	famous_u2pz_1399_200_006636370_1	842,426	321,740	0.3819