Task 12816542

Name	hadcm3n_p0k7_1900_40_007218311_0
Workunit	7416551
Created	26 Apr 2011, 15:14:37 UTC
Sent	5 May 2011, 9:44:58 UTC
Report deadline	4 Aug 2011, 17:12:09 UTC
Received	14 May 2011, 18:15:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1147579
Run time	3 days 0 hours 10 min 57 sec
CPU time	2 days 15 hours 16 min 31 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7016, iMonCtr=1 Model crash detected, will try to restart... 15:38:24 (6264): No heartbeat from core client for 30 sec - exiting 15:38:26 (6264): No heartbeat from core client for 30 sec - exiting 15:38:27 (6264): No heartbeat from core client for 30 sec - exiting 15:38:28 (6264): No heartbeat from core client for 30 sec - exiting 15:38:29 (6264): No heartbeat from core client for 30 sec - exiting 15:38:30 (6264): No heartbeat from core client for 30 sec - exiting 15:38:31 (6264): No heartbeat from core client for 30 sec - exiting 15:38:32 (6264): No heartbeat from core client for 30 sec - exiting 15:38:33 (6264): No heartbeat from core client for 30 sec - exiting 15:38:34 (6264): No heartbeat from core client for 30 sec - exiting 15:38:35 (6264): No heartbeat from core client for 30 sec - exiting 15:38:36 (6264): No heartbeat from core client for 30 sec - exiting 15:38:37 (6264): No heartbeat from core client for 30 sec - exiting 15:38:38 (6264): No heartbeat from core client for 30 sec - exiting 15:38:39 (6264): No heartbeat from core client for 30 sec - exiting 15:38:40 (6264): No heartbeat from core client for 30 sec - exiting 15:38:41 (6264): No heartbeat from core client for 30 sec - exiting 15:38:42 (6264): No heartbeat from core client for 30 sec - exiting 15:38:43 (6264): No heartbeat from core client for 30 sec - exiting 15:38:44 (6264): No heartbeat from core client for 30 sec - exiting 15:38:45 (6264): No heartbeat from core client for 30 sec - exiting 15:38:46 (6264): No heartbeat from core client for 30 sec - exiting 15:38:47 (6264): No heartbeat from core client for 30 sec - exiting 15:38:48 (6264): No heartbeat from core client for 30 sec - exiting 15:38:49 (6264): No heartbeat from core client for 30 sec - exiting 15:38:50 (6264): No heartbeat from core client for 30 sec - exiting 15:38:51 (6264): No heartbeat from core client for 30 sec - exiting 15:38:52 (6264): No heartbeat from core client for 30 sec - exiting 15:38:53 (6264): No heartbeat from core client for 30 sec - exiting 15:38:54 (6264): No heartbeat from core client for 30 sec - exiting 15:38:55 (6264): No heartbeat from core client for 30 sec - exiting 15:38:56 (6264): No heartbeat from core client for 30 sec - exiting 15:38:57 (6264): No heartbeat from core client for 30 sec - exiting 15:38:58 (6264): No heartbeat from core client for 30 sec - exiting 15:38:59 (6264): No heartbeat from core client for 30 sec - exiting 15:39:00 (6264): No heartbeat from core client for 30 sec - exiting 15:39:01 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:39:02 (6264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:22:08 (3428): No heartbeat from core client for 30 sec - exiting 18:22:09 (3428): No heartbeat from core client for 30 sec - exiting 18:22:10 (3428): No heartbeat from core client for 30 sec - exiting 18:22:11 (3428): No heartbeat from core client for 30 sec - exiting 18:22:12 (3428): No heartbeat from core client for 30 sec - exiting 18:22:13 (3428): No heartbeat from core client for 30 sec - exiting 18:22:14 (3428): No heartbeat from core client for 30 sec - exiting 18:22:15 (3428): No heartbeat from core client for 30 sec - exiting 18:22:16 (3428): No heartbeat from core client for 30 sec - exiting 18:22:17 (3428): No heartbeat from core client for 30 sec - exiting 18:22:18 (3428): No heartbeat from core client for 30 sec - exiting 18:22:19 (3428): No heartbeat from core client for 30 sec - exiting 18:22:20 (3428): No heartbeat from core client for 30 sec - exiting 18:22:21 (3428): No heartbeat from core client for 30 sec - exiting 18:22:22 (3428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... 08:35:13 (6640): No heartbeat from core client for 30 sec - exiting 08:35:14 (6640): No heartbeat from core client for 30 sec - exiting 08:35:15 (6640): No heartbeat from core client for 30 sec - exiting 08:35:16 (6640): No heartbeat from core client for 30 sec - exiting 08:35:17 (6640): No heartbeat from core client for 30 sec - exiting 08:35:18 (6640): No heartbeat from core client for 30 sec - exiting 08:35:19 (6640): No heartbeat from core client for 30 sec - exiting 08:35:20 (6640): No heartbeat from core client for 30 sec - exiting 08:35:21 (6640): No heartbeat from core client for 30 sec - exiting 08:35:22 (6640): No heartbeat from core client for 30 sec - exiting 08:35:23 (6640): No heartbeat from core client for 30 sec - exiting 08:35:24 (6640): No heartbeat from core client for 30 sec - exiting 08:35:25 (6640): No heartbeat from core client for 30 sec - exiting 08:35:26 (6640): No heartbeat from core client for 30 sec - exiting 08:35:27 (6640): No heartbeat from core client for 30 sec - exiting 08:35:28 (6640): No heartbeat from core client for 30 sec - exiting 08:35:29 (6640): No heartbeat from core client for 30 sec - exiting 08:35:30 (6640): No heartbeat from core client for 30 sec - exiting 08:35:31 (6640): No heartbeat from core client for 30 sec - exiting 08:35:32 (6640): No heartbeat from core client for 30 sec - exiting 08:35:33 (6640): No heartbeat from core client for 30 sec - exiting 08:35:34 (6640): No heartbeat from core client for 30 sec - exiting 08:35:35 (6640): No heartbeat from core client for 30 sec - exiting 08:35:36 (6640): No heartbeat from core client for 30 sec - exiting 08:35:37 (6640): No heartbeat from core client for 30 sec - exiting 08:35:38 (6640): No heartbeat from core client for 30 sec - exiting 08:35:39 (6640): No heartbeat from core client for 30 sec - exiting 08:35:40 (6640): No heartbeat from core client for 30 sec - exiting 08:35:41 (6640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:20:22 (4156): No heartbeat from core client for 30 sec - exiting 08:20:23 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5412, iMonCtr=1 Model crash detected, will try to restart... 08:24:37 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:13:04 (4304): No heartbeat from core client for 30 sec - exiting 11:13:05 (4304): No heartbeat from core client for 30 sec - exiting 11:13:06 (4304): No heartbeat from core client for 30 sec - exiting 11:13:07 (4304): No heartbeat from core client for 30 sec - exiting 11:13:08 (4304): No heartbeat from core client for 30 sec - exiting 11:13:09 (4304): No heartbeat from core client for 30 sec - exiting 11:13:10 (4304): No heartbeat from core client for 30 sec - exiting 11:13:11 (4304): No heartbeat from core client for 30 sec - exiting 11:13:12 (4304): No heartbeat from core client for 30 sec - exiting 11:13:13 (4304): No heartbeat from core client for 30 sec - exiting 11:13:14 (4304): No heartbeat from core client for 30 sec - exiting 11:13:15 (4304): No heartbeat from core client for 30 sec - exiting 11:13:16 (4304): No heartbeat from core client for 30 sec - exiting 11:13:17 (4304): No heartbeat from core client for 30 sec - exiting 11:13:18 (4304): No heartbeat from core client for 30 sec - exiting 11:13:19 (4304): No heartbeat from core client for 30 sec - exiting 11:13:20 (4304): No heartbeat from core client for 30 sec - exiting 11:13:21 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3320, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 May 2011 13:01:30	1147579	12816542	hadcm3n_p0k7_1900_40_007218311_0	103,680	198,793	1.9174
12 May 2011 19:37:14	1147579	12816542	hadcm3n_p0k7_1900_40_007218311_0	77,760	149,363	1.9208
11 May 2011 19:55:43	1147579	12816542	hadcm3n_p0k7_1900_40_007218311_0	51,840	100,854	1.9455
10 May 2011 12:08:48	1147579	12816542	hadcm3n_p0k7_1900_40_007218311_0	25,920	50,992	1.9673