Task 16012896

Name	hadcm3n_3ixd_1980_40_008364307_2
Workunit	8515166
Created	12 Sep 2013, 4:30:33 UTC
Sent	12 Sep 2013, 4:42:56 UTC
Report deadline	12 Dec 2013, 12:10:07 UTC
Received	28 Sep 2013, 3:20:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291838
Run time	15 days 7 hours 22 min 31 sec
CPU time	14 days 3 hours 47 min 34 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:10:08 (2704): No heartbeat from core client for 30 sec - exiting 07:10:09 (2704): No heartbeat from core client for 30 sec - exiting 07:10:10 (2704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:52:40 (11404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:38 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:39 (3540): No heartbeat from core client for 30 sec - exiting 20:10:02 (3492): No heartbeat from core client for 30 sec - exiting 20:10:03 (3492): No heartbeat from core client for 30 sec - exiting 20:10:04 (3492): No heartbeat from core client for 30 sec - exiting 20:10:05 (3492): No heartbeat from core client for 30 sec - exiting 20:10:06 (3492): No heartbeat from core client for 30 sec - exiting 20:10:07 (3492): No heartbeat from core client for 30 sec - exiting 20:10:08 (3492): No heartbeat from core client for 30 sec - exiting 20:10:09 (3492): No heartbeat from core client for 30 sec - exiting 20:10:10 (3492): No heartbeat from core client for 30 sec - exiting 20:10:11 (3492): No heartbeat from core client for 30 sec - exiting 20:10:12 (3492): No heartbeat from core client for 30 sec - exiting 20:10:13 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2552, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2552, iMonCtr=1 Model crash detected, will try to restart... 21:10:24 (2552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Sep 2013 16:12:03	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	648,000	1,200,962	1.8533
26 Sep 2013 17:31:15	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	622,080	1,154,416	1.8557
26 Sep 2013 03:55:12	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	596,160	1,108,641	1.8596
25 Sep 2013 14:40:41	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	570,240	1,063,007	1.8641
25 Sep 2013 09:26:50	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	544,320	1,018,021	1.8703
25 Sep 2013 09:14:10	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	518,400	972,926	1.8768
23 Sep 2013 23:59:33	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	492,480	928,376	1.8851
23 Sep 2013 11:27:06	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	466,560	881,018	1.8883
23 Sep 2013 11:27:06	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	440,640	833,886	1.8924
23 Sep 2013 11:27:06	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	414,720	783,574	1.8894
21 Sep 2013 12:41:42	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	388,800	730,638	1.8792
20 Sep 2013 21:16:50	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	362,880	682,001	1.8794
20 Sep 2013 06:13:22	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	336,960	630,071	1.8699
19 Sep 2013 11:48:04	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	311,040	578,856	1.8610
18 Sep 2013 20:13:53	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	285,120	530,107	1.8592
18 Sep 2013 06:41:53	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	259,200	484,320	1.8685
17 Sep 2013 18:03:02	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	233,280	439,871	1.8856
17 Sep 2013 05:27:06	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	207,360	395,868	1.9091
16 Sep 2013 16:03:55	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	181,440	351,469	1.9371
15 Sep 2013 21:57:48	1291838	16012896	hadcm3n_3ixd_1980_40_008364307_2	155,520	290,583	1.8685