Task 15820682

Name	hadcm3n_o1yh_1980_40_008385403_0
Workunit	8536262
Created	3 Jun 2013, 3:05:04 UTC
Sent	3 Jun 2013, 3:19:13 UTC
Report deadline	2 Sep 2013, 10:46:24 UTC
Received	18 Jul 2013, 14:39:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1259079
Run time	12 days 6 hours 2 min 1 sec
CPU time	12 days 4 hours 10 min 25 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	3.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 04:14:34 (8424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:36 (8424): No heartbeat from core client for 30 sec - exiting 04:14:38 (8424): No heartbeat from core client for 30 sec - exiting 04:16:21 (8992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:16:22 (8992): No heartbeat from core client for 30 sec - exiting 04:16:23 (8992): No heartbeat from core client for 30 sec - exiting 04:16:24 (8992): No heartbeat from core client for 30 sec - exiting 04:16:25 (8992): No heartbeat from core client for 30 sec - exiting 04:16:26 (8992): No heartbeat from core client for 30 sec - exiting 04:16:27 (8992): No heartbeat from core client for 30 sec - exiting 04:16:28 (8992): No heartbeat from core client for 30 sec - exiting 04:16:29 (8992): No heartbeat from core client for 30 sec - exiting 04:16:30 (8992): No heartbeat from core client for 30 sec - exiting 04:16:31 (8992): No heartbeat from core client for 30 sec - exiting 06:21:31 (7324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:34 (7324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:54:10 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:12 (5716): No heartbeat from core client for 30 sec - exiting 03:55:54 (7360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:49:45 (2096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:31:53 (3220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:56 (3220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:32:52 (13696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:18 (12560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:00:13 (6724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:00:14 (6724): No heartbeat from core client for 30 sec - exiting 00:00:16 (6724): No heartbeat from core client for 30 sec - exiting 00:30:15 (8524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:30:17 (8524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:48:00 (3332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:49:18 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:19 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:20 (3376): No heartbeat from core client for 30 sec - exiting 11:52:21 (3376): No heartbeat from core client for 30 sec - exiting 11:52:22 (3376): No heartbeat from core client for 30 sec - exiting 11:52:23 (3376): No heartbeat from core client for 30 sec - exiting 11:52:24 (3376): No heartbeat from core client for 30 sec - exiting 11:52:25 (3376): No heartbeat from core client for 30 sec - exiting 11:52:26 (3376): No heartbeat from core client for 30 sec - exiting 11:52:27 (3376): No heartbeat from core client for 30 sec - exiting 11:52:28 (3376): No heartbeat from core client for 30 sec - exiting 11:52:29 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:02:29 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:20 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:07:37 (4468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:07:38 (4468): No heartbeat from core client for 30 sec - exiting 17:07:40 (4468): No heartbeat from core client for 30 sec - exiting 17:07:41 (4468): No heartbeat from core client for 30 sec - exiting 17:07:43 (4468): No heartbeat from core client for 30 sec - exiting 17:07:45 (4468): No heartbeat from core client for 30 sec - exiting 17:07:47 (4468): No heartbeat from core client for 30 sec - exiting 18:19:33 (6908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:34 (6908): No heartbeat from core client for 30 sec - exiting 18:27:42 (7244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:44 (7244): No heartbeat from core client for 30 sec - exiting 18:56:17 (7984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:20 (7984): No heartbeat from core client for 30 sec - exiting 18:57:47 (6808): No heartbeat from core client for 30 sec - exiting 18:57:49 (6808): No heartbeat from core client for 30 sec - exiting 18:57:50 (6808): No heartbeat from core client for 30 sec - exiting 18:57:52 (6808): No heartbeat from core client for 30 sec - exiting 18:57:54 (6808): No heartbeat from core client for 30 sec - exiting 18:57:55 (6808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:00:08 (7112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:00:09 (7112): No heartbeat from core client for 30 sec - exiting 03:00:12 (7112): No heartbeat from core client for 30 sec - exiting 04:05:10 (7148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:12 (7148): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6592, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	984,960	1,036,034	1.0519
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	959,040	1,009,976	1.0531
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	933,120	983,922	1.0544
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	907,200	957,985	1.0560
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	881,280	932,307	1.0579
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	855,360	906,375	1.0596
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	829,440	880,528	1.0616
23 Jul 2013 17:26:43	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	803,520	854,910	1.0640
23 Jul 2013 17:26:42	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	777,600	829,118	1.0663
23 Jul 2013 17:26:42	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	751,680	803,306	1.0687
23 Jul 2013 17:26:42	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	725,760	777,458	1.0712
23 Jul 2013 17:26:42	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	699,840	751,794	1.0742
23 Jul 2013 17:26:42	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	673,920	725,885	1.0771
14 Jun 2013 07:11:54	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	648,000	698,951	1.0786
13 Jun 2013 22:51:55	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	622,080	668,962	1.0754
13 Jun 2013 06:07:48	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	596,160	639,483	1.0727
12 Jun 2013 19:56:34	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	570,240	610,231	1.0701
12 Jun 2013 05:29:38	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	544,320	580,915	1.0672
11 Jun 2013 13:00:03	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	518,400	551,397	1.0637
11 Jun 2013 04:46:33	1259079	15820682	hadcm3n_o1yh_1980_40_008385403_0	492,480	521,874	1.0597