Task 13352600

Name	hadcm3n_t5v9_1940_40_007443888_3
Workunit	7641391
Created	9 Sep 2011, 10:05:26 UTC
Sent	9 Sep 2011, 11:00:27 UTC
Report deadline	9 Dec 2011, 18:27:38 UTC
Received	21 Sep 2011, 12:34:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1168646
Run time	8 days 9 hours 16 min 12 sec
CPU time	6 days 8 hours 2 min 5 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	2.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3692, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 01:14:52 (4784): No heartbeat from core client for 30 sec - exiting 01:14:59 (4784): No heartbeat from core client for 30 sec - exiting 01:15:00 (4784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:01 (4784): No heartbeat from core client for 30 sec - exiting 01:15:02 (4784): No heartbeat from core client for 30 sec - exiting 01:15:04 (4784): No heartbeat from core client for 30 sec - exiting 01:15:05 (4784): No heartbeat from core client for 30 sec - exiting 01:15:07 (4784): No heartbeat from core client for 30 sec - exiting 01:15:08 (4784): No heartbeat from core client for 30 sec - exiting 01:15:09 (4784): No heartbeat from core client for 30 sec - exiting 01:15:10 (4784): No heartbeat from core client for 30 sec - exiting 01:15:13 (4784): No heartbeat from core client for 30 sec - exiting 01:15:15 (4784): No heartbeat from core client for 30 sec - exiting 01:15:16 (4784): No heartbeat from core client for 30 sec - exiting 01:01:40 (3976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:41 (3976): No heartbeat from core client for 30 sec - exiting 01:01:42 (3976): No heartbeat from core client for 30 sec - exiting 01:01:45 (3976): No heartbeat from core client for 30 sec - exiting 01:01:47 (3976): No heartbeat from core client for 30 sec - exiting 01:01:49 (3976): No heartbeat from core client for 30 sec - exiting 01:01:50 (3976): No heartbeat from core client for 30 sec - exiting 01:01:52 (3976): No heartbeat from core client for 30 sec - exiting 01:01:53 (3976): No heartbeat from core client for 30 sec - exiting 01:01:54 (3976): No heartbeat from core client for 30 sec - exiting 01:01:55 (3976): No heartbeat from core client for 30 sec - exiting 01:01:56 (3976): No heartbeat from core client for 30 sec - exiting 01:01:58 (3976): No heartbeat from core client for 30 sec - exiting 01:02:00 (3976): No heartbeat from core client for 30 sec - exiting 01:03:20 (5552): No heartbeat from core client for 30 sec - exiting 01:03:42 (5552): No heartbeat from core client for 30 sec - exiting 01:03:43 (5552): No heartbeat from core client for 30 sec - exiting 01:03:44 (5552): No heartbeat from core client for 30 sec - exiting 01:03:46 (5552): No heartbeat from core client for 30 sec - exiting 01:03:47 (5552): No heartbeat from core client for 30 sec - exiting 01:03:48 (5552): No heartbeat from core client for 30 sec - exiting 01:03:49 (5552): No heartbeat from core client for 30 sec - exiting 01:03:50 (5552): No heartbeat from core client for 30 sec - exiting 01:03:51 (5552): No heartbeat from core client for 30 sec - exiting 01:03:53 (5552): No heartbeat from core client for 30 sec - exiting 01:03:54 (5552): No heartbeat from core client for 30 sec - exiting 01:03:55 (5552): No heartbeat from core client for 30 sec - exiting 01:03:56 (5552): No heartbeat from core client for 30 sec - exiting 01:03:57 (5552): No heartbeat from core client for 30 sec - exiting 01:03:59 (5552): No heartbeat from core client for 30 sec - exiting 01:04:01 (5552): No heartbeat from core client for 30 sec - exiting 01:04:03 (5552): No heartbeat from core client for 30 sec - exiting 01:04:04 (5552): No heartbeat from core client for 30 sec - exiting 01:04:05 (5552): No heartbeat from core client for 30 sec - exiting 01:04:06 (5552): No heartbeat from core client for 30 sec - exiting 01:04:07 (5552): No heartbeat from core client for 30 sec - exiting 01:04:09 (5552): No heartbeat from core client for 30 sec - exiting 01:04:11 (5552): No heartbeat from core client for 30 sec - exiting 01:04:14 (5552): No heartbeat from core client for 30 sec - exiting 01:04:15 (5552): No heartbeat from core client for 30 sec - exiting 01:04:17 (5552): No heartbeat from core client for 30 sec - exiting 01:04:19 (5552): No heartbeat from core client for 30 sec - exiting 01:04:21 (5552): No heartbeat from core client for 30 sec - exiting 01:04:22 (5552): No heartbeat from core client for 30 sec - exiting 01:04:24 (5552): No heartbeat from core client for 30 sec - exiting 01:04:31 (5552): No heartbeat from core client for 30 sec - exiting 01:04:32 (5552): No heartbeat from core client for 30 sec - exiting 01:04:33 (5552): No heartbeat from core client for 30 sec - exiting 01:04:34 (5552): No heartbeat from core client for 30 sec - exiting 01:04:36 (5552): No heartbeat from core client for 30 sec - exiting 01:04:38 (5552): No heartbeat from core client for 30 sec - exiting 01:04:39 (5552): No heartbeat from core client for 30 sec - exiting 01:04:41 (5552): No heartbeat from core client for 30 sec - exiting 01:04:42 (5552): No heartbeat from core client for 30 sec - exiting 01:04:43 (5552): No heartbeat from core client for 30 sec - exiting 01:04:44 (5552): No heartbeat from core client for 30 sec - exiting 01:04:45 (5552): No heartbeat from core client for 30 sec - exiting 01:04:47 (5552): No heartbeat from core client for 30 sec - exiting 01:04:48 (5552): No heartbeat from core client for 30 sec - exiting 01:04:49 (5552): No heartbeat from core client for 30 sec - exiting 01:04:51 (5552): No heartbeat from core client for 30 sec - exiting 01:04:52 (5552): No heartbeat from core client for 30 sec - exiting 01:04:54 (5552): No heartbeat from core client for 30 sec - exiting 01:04:56 (5552): No heartbeat from core client for 30 sec - exiting 01:04:58 (5552): No heartbeat from core client for 30 sec - exiting 01:05:00 (5552): No heartbeat from core client for 30 sec - exiting 01:05:01 (5552): No heartbeat from core client for 30 sec - exiting 01:05:03 (5552): No heartbeat from core client for 30 sec - exiting 01:05:04 (5552): No heartbeat from core client for 30 sec - exiting 01:05:05 (5552): No heartbeat from core client for 30 sec - exiting 01:05:07 (5552): No heartbeat from core client for 30 sec - exiting 01:05:08 (5552): No heartbeat from core client for 30 sec - exiting 01:05:10 (5552): No heartbeat from core client for 30 sec - exiting 01:05:11 (5552): No heartbeat from core client for 30 sec - exiting 01:05:12 (5552): No heartbeat from core client for 30 sec - exiting 01:05:13 (5552): No heartbeat from core client for 30 sec - exiting 01:05:14 (5552): No heartbeat from core client for 30 sec - exiting 01:05:15 (5552): No heartbeat from core client for 30 sec - exiting 01:05:16 (5552): No heartbeat from core client for 30 sec - exiting 01:05:17 (5552): No heartbeat from core client for 30 sec - exiting 01:05:19 (5552): No heartbeat from core client for 30 sec - exiting 01:05:20 (5552): No heartbeat from core client for 30 sec - exiting 01:05:23 (5552): No heartbeat from core client for 30 sec - exiting 01:05:24 (5552): No heartbeat from core client for 30 sec - exiting 01:05:25 (5552): No heartbeat from core client for 30 sec - exiting 01:05:26 (5552): No heartbeat from core client for 30 sec - exiting 01:05:27 (5552): No heartbeat from core client for 30 sec - exiting 01:05:28 (5552): No heartbeat from core client for 30 sec - exiting 01:05:29 (5552): No heartbeat from core client for 30 sec - exiting 01:05:30 (5552): No heartbeat from core client for 30 sec - exiting 01:05:32 (5552): No heartbeat from core client for 30 sec - exiting 01:05:33 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:17 (5552): No heartbeat from core client for 30 sec - exiting 01:06:18 (5552): No heartbeat from core client for 30 sec - exiting 01:06:19 (5552): No heartbeat from core client for 30 sec - exiting 01:10:33 (3836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:11:39 (1356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:11:56 (1356): No heartbeat from core client for 30 sec - exiting 01:12:51 (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:01 (5628): No heartbeat from core client for 30 sec - exiting 01:13:45 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:50 (4708): No heartbeat from core client for 30 sec - exiting 01:14:57 (5840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:14 (3212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:52 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:19:04 (3640): No heartbeat from core client for 30 sec - exiting 01:20:28 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:19 (3380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:23:30 (1908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:28:53 (3864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:30:48 (3232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:33:32 (1836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:44 (5840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:39:36 (3112): No heartbeat from core client for 30 sec - exiting 01:39:42 (3112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:41:46 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:44:33 (4448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:47:12 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:48:45 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:48:56 (5504): No heartbeat from core client for 30 sec - exiting 01:48:57 (5504): No heartbeat from core client for 30 sec - exiting 01:48:58 (5504): No heartbeat from core client for 30 sec - exiting 01:48:59 (5504): No heartbeat from core client for 30 sec - exiting 01:49:01 (5504): No heartbeat from core client for 30 sec - exiting 01:49:02 (5504): No heartbeat from core client for 30 sec - exiting 01:50:44 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:52:33 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:35 (5836): No heartbeat from core client for 30 sec - exiting 01:55:45 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:25 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:54 (2776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:55 (2776): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Sep 2011 15:28:11	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	285,120	523,943	1.8376
18 Sep 2011 02:21:08	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	259,200	477,072	1.8406
17 Sep 2011 12:42:21	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	233,280	429,369	1.8406
16 Sep 2011 23:20:31	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	207,360	381,463	1.8396
16 Sep 2011 09:49:24	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	181,440	333,681	1.8391
15 Sep 2011 13:38:13	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	155,520	285,508	1.8358
15 Sep 2011 00:04:39	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	129,600	237,803	1.8349
14 Sep 2011 10:37:08	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	103,680	189,960	1.8322
13 Sep 2011 21:13:47	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	77,760	142,473	1.8322
13 Sep 2011 07:53:21	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	51,840	95,034	1.8332
12 Sep 2011 18:30:56	1168646	13352600	hadcm3n_t5v9_1940_40_007443888_3	25,920	47,610	1.8368