Task 13263866

Name	hadcm3n_t142_1980_40_007411591_1
Workunit	7609221
Created	16 Aug 2011, 20:54:42 UTC
Sent	16 Aug 2011, 21:09:13 UTC
Report deadline	16 Nov 2011, 4:36:24 UTC
Received	7 Oct 2011, 0:14:45 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	879185
Run time	18 days 11 hours 8 min 43 sec
CPU time	14 days 13 hours 15 min 51 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:10:33 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:36 (4944): No heartbeat from core client for 30 sec - exiting 00:10:37 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:05:46 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:53 (3992): No heartbeat from core client for 30 sec - exiting 00:05:55 (3992): No heartbeat from core client for 30 sec - exiting 00:05:57 (3992): No heartbeat from core client for 30 sec - exiting 00:05:58 (3992): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 00:04:38 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:46 (5808): No heartbeat from core client for 30 sec - exiting 00:04:48 (5808): No heartbeat from core client for 30 sec - exiting 00:04:50 (5808): No heartbeat from core client for 30 sec - exiting 00:04:51 (5808): No heartbeat from core client for 30 sec - exiting 00:04:52 (5808): No heartbeat from core client for 30 sec - exiting 00:04:53 (5808): No heartbeat from core client for 30 sec - exiting 00:04:54 (5808): No heartbeat from core client for 30 sec - exiting 00:04:55 (5808): No heartbeat from core client for 30 sec - exiting 00:04:56 (5808): No heartbeat from core client for 30 sec - exiting 00:04:57 (5808): No heartbeat from core client for 30 sec - exiting 00:04:58 (5808): No heartbeat from core client for 30 sec - exiting 00:04:59 (5808): No heartbeat from core client for 30 sec - exiting 00:05:00 (5808): No heartbeat from core client for 30 sec - exiting 00:05:02 (5808): No heartbeat from core client for 30 sec - exiting 00:05:03 (5808): No heartbeat from core client for 30 sec - exiting 00:05:04 (5808): No heartbeat from core client for 30 sec - exiting 00:05:05 (5808): No heartbeat from core client for 30 sec - exiting 00:05:06 (5808): No heartbeat from core client for 30 sec - exiting 00:05:08 (5808): No heartbeat from core client for 30 sec - exiting 00:04:58 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:05:34 (5856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:39 (5856): No heartbeat from core client for 30 sec - exiting 00:05:41 (5856): No heartbeat from core client for 30 sec - exiting 00:05:42 (5856): No heartbeat from core client for 30 sec - exiting 00:05:43 (5856): No heartbeat from core client for 30 sec - exiting 00:05:44 (5856): No heartbeat from core client for 30 sec - exiting 00:05:45 (5856): No heartbeat from core client for 30 sec - exiting 00:05:47 (5856): No heartbeat from core client for 30 sec - exiting 00:05:48 (5856): No heartbeat from core client for 30 sec - exiting 00:05:49 (5856): No heartbeat from core client for 30 sec - exiting 00:05:50 (5856): No heartbeat from core client for 30 sec - exiting 00:05:52 (5856): No heartbeat from core client for 30 sec - exiting 00:05:53 (5856): No heartbeat from core client for 30 sec - exiting 00:05:55 (5856): No heartbeat from core client for 30 sec - exiting 00:05:56 (5856): No heartbeat from core client for 30 sec - exiting 00:05:57 (5856): No heartbeat from core client for 30 sec - exiting 00:05:58 (5856): No heartbeat from core client for 30 sec - exiting 00:06:00 (5856): No heartbeat from core client for 30 sec - exiting 00:05:04 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:08 (3940): No heartbeat from core client for 30 sec - exiting 00:05:09 (3940): No heartbeat from core client for 30 sec - exiting 00:05:10 (3940): No heartbeat from core client for 30 sec - exiting 00:05:11 (3940): No heartbeat from core client for 30 sec - exiting 00:05:13 (3940): No heartbeat from core client for 30 sec - exiting 00:05:14 (3940): No heartbeat from core client for 30 sec - exiting 00:05:15 (3940): No heartbeat from core client for 30 sec - exiting 00:05:16 (3940): No heartbeat from core client for 30 sec - exiting 00:05:18 (3940): No heartbeat from core client for 30 sec - exiting 00:05:19 (3940): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Oct 2011 19:34:26	879185	13263866	hadcm3n_t142_1980_40_007411591_1	881,280	1,246,465	1.4144
06 Oct 2011 05:50:10	879185	13263866	hadcm3n_t142_1980_40_007411591_1	855,360	1,209,624	1.4142
05 Oct 2011 17:04:22	879185	13263866	hadcm3n_t142_1980_40_007411591_1	829,440	1,172,695	1.4138
05 Oct 2011 04:30:36	879185	13263866	hadcm3n_t142_1980_40_007411591_1	803,520	1,135,833	1.4136
04 Oct 2011 15:22:59	879185	13263866	hadcm3n_t142_1980_40_007411591_1	777,600	1,098,941	1.4132
03 Oct 2011 18:06:29	879185	13263866	hadcm3n_t142_1980_40_007411591_1	751,680	1,062,423	1.4134
02 Oct 2011 22:27:55	879185	13263866	hadcm3n_t142_1980_40_007411591_1	725,760	1,026,756	1.4147
02 Oct 2011 01:40:11	879185	13263866	hadcm3n_t142_1980_40_007411591_1	699,840	990,697	1.4156
01 Oct 2011 11:56:10	879185	13263866	hadcm3n_t142_1980_40_007411591_1	673,920	954,226	1.4159
30 Sep 2011 21:27:37	879185	13263866	hadcm3n_t142_1980_40_007411591_1	648,000	917,726	1.4162
30 Sep 2011 08:10:20	879185	13263866	hadcm3n_t142_1980_40_007411591_1	622,080	880,821	1.4159
29 Sep 2011 19:18:06	879185	13263866	hadcm3n_t142_1980_40_007411591_1	596,160	843,949	1.4156
29 Sep 2011 05:45:40	879185	13263866	hadcm3n_t142_1980_40_007411591_1	570,240	807,065	1.4153
28 Sep 2011 16:42:15	879185	13263866	hadcm3n_t142_1980_40_007411591_1	544,320	770,168	1.4149
28 Sep 2011 03:20:10	879185	13263866	hadcm3n_t142_1980_40_007411591_1	518,400	733,257	1.4145
27 Sep 2011 12:55:51	879185	13263866	hadcm3n_t142_1980_40_007411591_1	492,480	696,371	1.4140
26 Sep 2011 21:45:19	879185	13263866	hadcm3n_t142_1980_40_007411591_1	466,560	659,769	1.4141
26 Sep 2011 08:29:36	879185	13263866	hadcm3n_t142_1980_40_007411591_1	440,640	623,008	1.4139
25 Sep 2011 19:50:14	879185	13263866	hadcm3n_t142_1980_40_007411591_1	414,720	586,146	1.4134
25 Sep 2011 05:48:54	879185	13263866	hadcm3n_t142_1980_40_007411591_1	388,800	549,264	1.4127