Task 15549320

Name	hadcm3n_n5ab_1880_40_008286427_0
Workunit	8437562
Created	17 Jan 2013, 19:50:05 UTC
Sent	17 Jan 2013, 20:20:43 UTC
Report deadline	19 Apr 2013, 3:47:54 UTC
Received	17 Feb 2013, 9:47:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1001286
Run time	7 days 2 hours 47 min 25 sec
CPU time	5 days 3 hours 34 min 22 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 09:39:00 (2572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:39:01 (2572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 09:39:34 (1452): No heartbeat from core client for 30 sec - exiting 09:39:35 (1452): No heartbeat from core client for 30 sec - exiting 09:39:36 (1452): No heartbeat from core client for 30 sec - exiting 09:39:37 (1452): No heartbeat from core client for 30 sec - exiting 09:39:38 (1452): No heartbeat from core client for 30 sec - exiting 09:39:40 (1452): No heartbeat from core client for 30 sec - exiting 09:39:41 (1452): No heartbeat from core client for 30 sec - exiting 09:39:42 (1452): No heartbeat from core client for 30 sec - exiting 09:39:43 (1452): No heartbeat from core client for 30 sec - exiting 09:39:44 (1452): No heartbeat from core client for 30 sec - exiting 09:39:45 (1452): No heartbeat from core client for 30 sec - exiting 09:39:46 (1452): No heartbeat from core client for 30 sec - exiting 09:39:47 (1452): No heartbeat from core client for 30 sec - exiting 09:39:48 (1452): No heartbeat from core client for 30 sec - exiting 09:39:49 (1452): No heartbeat from core client for 30 sec - exiting 09:39:51 (1452): No heartbeat from core client for 30 sec - exiting 09:39:52 (1452): No heartbeat from core client for 30 sec - exiting 09:39:53 (1452): No heartbeat from core client for 30 sec - exiting 09:39:54 (1452): No heartbeat from core client for 30 sec - exiting 09:39:55 (1452): No heartbeat from core client for 30 sec - exiting 09:39:56 (1452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:30:20 (2912): No heartbeat from core client for 30 sec - exiting 09:30:21 (2912): No heartbeat from core client for 30 sec - exiting 09:30:22 (2912): No heartbeat from core client for 30 sec - exiting 09:30:23 (2912): No heartbeat from core client for 30 sec - exiting 09:30:25 (2912): No heartbeat from core client for 30 sec - exiting 09:30:26 (2912): No heartbeat from core client for 30 sec - exiting 09:30:27 (2912): No heartbeat from core client for 30 sec - exiting 09:30:28 (2912): No heartbeat from core client for 30 sec - exiting 09:30:29 (2912): No heartbeat from core client for 30 sec - exiting 09:30:30 (2912): No heartbeat from core client for 30 sec - exiting 09:30:31 (2912): No heartbeat from core client for 30 sec - exiting 09:30:32 (2912): No heartbeat from core client for 30 sec - exiting 09:30:33 (2912): No heartbeat from core client for 30 sec - exiting 09:30:34 (2912): No heartbeat from core client for 30 sec - exiting 09:30:35 (2912): No heartbeat from core client for 30 sec - exiting 09:30:37 (2912): No heartbeat from core client for 30 sec - exiting 09:30:38 (2912): No heartbeat from core client for 30 sec - exiting 09:30:39 (2912): No heartbeat from core client for 30 sec - exiting 09:30:40 (2912): No heartbeat from core client for 30 sec - exiting 09:30:41 (2912): No heartbeat from core client for 30 sec - exiting 09:30:42 (2912): No heartbeat from core client for 30 sec - exiting 09:30:43 (2912): No heartbeat from core client for 30 sec - exiting 09:30:44 (2912): No heartbeat from core client for 30 sec - exiting 09:30:45 (2912): No heartbeat from core client for 30 sec - exiting 09:30:46 (2912): No heartbeat from core client for 30 sec - exiting 09:30:48 (2912): No heartbeat from core client for 30 sec - exiting 09:30:49 (2912): No heartbeat from core client for 30 sec - exiting 09:30:50 (2912): No heartbeat from core client for 30 sec - exiting 09:30:51 (2912): No heartbeat from core client for 30 sec - exiting 09:30:52 (2912): No heartbeat from core client for 30 sec - exiting 09:30:53 (2912): No heartbeat from core client for 30 sec - exiting 09:30:54 (2912): No heartbeat from core client for 30 sec - exiting 09:30:55 (2912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:52 (3260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:37:50 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:37:51 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:50:08 (1912): No heartbeat from core client for 30 sec - exiting 14:50:10 (1912): No heartbeat from core client for 30 sec - exiting 14:50:11 (1912): No heartbeat from core client for 30 sec - exiting 14:50:12 (1912): No heartbeat from core client for 30 sec - exiting 14:50:13 (1912): No heartbeat from core client for 30 sec - exiting 14:50:14 (1912): No heartbeat from core client for 30 sec - exiting 14:50:15 (1912): No heartbeat from core client for 30 sec - exiting 14:50:16 (1912): No heartbeat from core client for 30 sec - exiting 14:50:17 (1912): No heartbeat from core client for 30 sec - exiting 14:50:18 (1912): No heartbeat from core client for 30 sec - exiting 14:50:19 (1912): No heartbeat from core client for 30 sec - exiting 14:50:20 (1912): No heartbeat from core client for 30 sec - exiting 14:50:22 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:47:52 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:38:10 (3356): No heartbeat from core client for 30 sec - exiting 10:38:11 (3356): No heartbeat from core client for 30 sec - exiting 10:38:12 (3356): No heartbeat from core client for 30 sec - exiting 10:38:13 (3356): No heartbeat from core client for 30 sec - exiting 10:38:14 (3356): No heartbeat from core client for 30 sec - exiting 10:38:15 (3356): No heartbeat from core client for 30 sec - exiting 10:38:16 (3356): No heartbeat from core client for 30 sec - exiting 10:38:18 (3356): No heartbeat from core client for 30 sec - exiting 10:38:19 (3356): No heartbeat from core client for 30 sec - exiting 10:38:20 (3356): No heartbeat from core client for 30 sec - exiting 10:38:21 (3356): No heartbeat from core client for 30 sec - exiting 10:38:22 (3356): No heartbeat from core client for 30 sec - exiting 10:38:23 (3356): No heartbeat from core client for 30 sec - exiting 10:38:24 (3356): No heartbeat from core client for 30 sec - exiting 10:38:25 (3356): No heartbeat from core client for 30 sec - exiting 10:38:26 (3356): No heartbeat from core client for 30 sec - exiting 10:38:27 (3356): No heartbeat from core client for 30 sec - exiting 10:38:28 (3356): No heartbeat from core client for 30 sec - exiting 10:38:30 (3356): No heartbeat from core client for 30 sec - exiting 10:38:31 (3356): No heartbeat from core client for 30 sec - exiting 10:38:32 (3356): No heartbeat from core client for 30 sec - exiting 10:38:33 (3356): No heartbeat from core client for 30 sec - exiting 10:38:34 (3356): No heartbeat from core client for 30 sec - exiting 10:38:35 (3356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:22:16 (3624): No heartbeat from core client for 30 sec - exiting 09:22:18 (3624): No heartbeat from core client for 30 sec - exiting 09:22:19 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:49:58 (580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:32:30 (3056): No heartbeat from core client for 30 sec - exiting 08:32:31 (3056): No heartbeat from core client for 30 sec - exiting 08:32:32 (3056): No heartbeat from core client for 30 sec - exiting 08:32:34 (3056): No heartbeat from core client for 30 sec - exiting 08:32:35 (3056): No heartbeat from core client for 30 sec - exiting 08:32:36 (3056): No heartbeat from core client for 30 sec - exiting 08:32:37 (3056): No heartbeat from core client for 30 sec - exiting 08:32:38 (3056): No heartbeat from core client for 30 sec - exiting 08:32:39 (3056): No heartbeat from core client for 30 sec - exiting 08:32:40 (3056): No heartbeat from core client for 30 sec - exiting 08:32:41 (3056): No heartbeat from core client for 30 sec - exiting 08:32:42 (3056): No heartbeat from core client for 30 sec - exiting 08:32:43 (3056): No heartbeat from core client for 30 sec - exiting 08:32:44 (3056): No heartbeat from core client for 30 sec - exiting 08:32:46 (3056): No heartbeat from core client for 30 sec - exiting 08:32:47 (3056): No heartbeat from core client for 30 sec - exiting 08:32:48 (3056): No heartbeat from core client for 30 sec - exiting 08:32:49 (3056): No heartbeat from core client for 30 sec - exiting 08:32:50 (3056): No heartbeat from core client for 30 sec - exiting 08:32:51 (3056): No heartbeat from core client for 30 sec - exiting 08:32:52 (3056): No heartbeat from core client for 30 sec - exiting 08:32:53 (3056): No heartbeat from core client for 30 sec - exiting 08:32:55 (3056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:27 (472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Feb 2013 03:00:09	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	233,280	401,932	1.7230
12 Feb 2013 09:59:54	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	207,360	357,210	1.7227
11 Feb 2013 07:40:57	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	181,440	312,020	1.7197
10 Feb 2013 13:10:10	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	155,520	266,564	1.7140
09 Feb 2013 10:37:15	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	129,600	222,701	1.7184
27 Jan 2013 20:46:25	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	103,680	178,174	1.7185
24 Jan 2013 22:28:09	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	77,760	133,335	1.7147
21 Jan 2013 16:15:57	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	51,840	89,142	1.7196
19 Jan 2013 10:15:26	1001286	15549320	hadcm3n_n5ab_1880_40_008286427_0	25,920	44,545	1.7186