Task 13139792

Name	hadcm3n_ykj8_1900_40_007359470_2
Workunit	7556900
Created	15 Jul 2011, 17:18:24 UTC
Sent	15 Jul 2011, 17:19:23 UTC
Report deadline	15 Oct 2011, 0:46:34 UTC
Received	29 Jul 2011, 14:57:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1147730
Run time	4 days 18 hours 51 min 14 sec
CPU time	4 days 9 hours 58 min 29 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:59:42 (3376): No heartbeat from core client for 30 sec - exiting 19:59:43 (3376): No heartbeat from core client for 30 sec - exiting 19:59:44 (3376): No heartbeat from core client for 30 sec - exiting 19:59:45 (3376): No heartbeat from core client for 30 sec - exiting 19:59:46 (3376): No heartbeat from core client for 30 sec - exiting 19:59:47 (3376): No heartbeat from core client for 30 sec - exiting 19:59:48 (3376): No heartbeat from core client for 30 sec - exiting 19:59:49 (3376): No heartbeat from core client for 30 sec - exiting 19:59:50 (3376): No heartbeat from core client for 30 sec - exiting 19:59:52 (3376): No heartbeat from core client for 30 sec - exiting 19:59:53 (3376): No heartbeat from core client for 30 sec - exiting 19:59:54 (3376): No heartbeat from core client for 30 sec - exiting 19:59:55 (3376): No heartbeat from core client for 30 sec - exiting 19:59:56 (3376): No heartbeat from core client for 30 sec - exiting 19:59:57 (3376): No heartbeat from core client for 30 sec - exiting 19:59:58 (3376): No heartbeat from core client for 30 sec - exiting 19:59:59 (3376): No heartbeat from core client for 30 sec - exiting 20:00:00 (3376): No heartbeat from core client for 30 sec - exiting 20:00:01 (3376): No heartbeat from core client for 30 sec - exiting 20:00:03 (3376): No heartbeat from core client for 30 sec - exiting 20:00:04 (3376): No heartbeat from core client for 30 sec - exiting 20:00:05 (3376): No heartbeat from core client for 30 sec - exiting 20:00:06 (3376): No heartbeat from core client for 30 sec - exiting 20:00:07 (3376): No heartbeat from core client for 30 sec - exiting 20:00:08 (3376): No heartbeat from core client for 30 sec - exiting 20:00:09 (3376): No heartbeat from core client for 30 sec - exiting 20:00:10 (3376): No heartbeat from core client for 30 sec - exiting 20:00:11 (3376): No heartbeat from core client for 30 sec - exiting 20:00:12 (3376): No heartbeat from core client for 30 sec - exiting 20:00:13 (3376): No heartbeat from core client for 30 sec - exiting 20:00:15 (3376): No heartbeat from core client for 30 sec - exiting 20:00:16 (3376): No heartbeat from core client for 30 sec - exiting 20:00:17 (3376): No heartbeat from core client for 30 sec - exiting 20:00:18 (3376): No heartbeat from core client for 30 sec - exiting 20:00:19 (3376): No heartbeat from core client for 30 sec - exiting 20:00:20 (3376): No heartbeat from core client for 30 sec - exiting 20:00:21 (3376): No heartbeat from core client for 30 sec - exiting 20:00:22 (3376): No heartbeat from core client for 30 sec - exiting 20:00:23 (3376): No heartbeat from core client for 30 sec - exiting 20:00:24 (3376): No heartbeat from core client for 30 sec - exiting 20:00:25 (3376): No heartbeat from core client for 30 sec - exiting 20:00:27 (3376): No heartbeat from core client for 30 sec - exiting 20:00:28 (3376): No heartbeat from core client for 30 sec - exiting 20:00:29 (3376): No heartbeat from core client for 30 sec - exiting 20:00:30 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:31 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:51:06 (4344): No heartbeat from core client for 30 sec - exiting 22:51:07 (4344): No heartbeat from core client for 30 sec - exiting 22:51:08 (4344): No heartbeat from core client for 30 sec - exiting 22:51:09 (4344): No heartbeat from core client for 30 sec - exiting 22:51:10 (4344): No heartbeat from core client for 30 sec - exiting 22:51:11 (4344): No heartbeat from core client for 30 sec - exiting 22:51:12 (4344): No heartbeat from core client for 30 sec - exiting 22:51:13 (4344): No heartbeat from core client for 30 sec - exiting 22:51:14 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:29:40 (136): No heartbeat from core client for 30 sec - exiting 01:29:41 (136): No heartbeat from core client for 30 sec - exiting 01:29:42 (136): No heartbeat from core client for 30 sec - exiting 01:29:43 (136): No heartbeat from core client for 30 sec - exiting 01:29:44 (136): No heartbeat from core client for 30 sec - exiting 01:29:45 (136): No heartbeat from core client for 30 sec - exiting 01:29:46 (136): No heartbeat from core client for 30 sec - exiting 01:29:47 (136): No heartbeat from core client for 30 sec - exiting 01:29:48 (136): No heartbeat from core client for 30 sec - exiting 01:29:49 (136): No heartbeat from core client for 30 sec - exiting 01:29:50 (136): No heartbeat from core client for 30 sec - exiting 01:29:51 (136): No heartbeat from core client for 30 sec - exiting 01:29:52 (136): No heartbeat from core client for 30 sec - exiting 01:29:53 (136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:30:51 (10232): No heartbeat from core client for 30 sec - exiting 02:30:52 (10232): No heartbeat from core client for 30 sec - exiting 02:30:53 (10232): No heartbeat from core client for 30 sec - exiting 02:30:54 (10232): No heartbeat from core client for 30 sec - exiting 02:30:55 (10232): No heartbeat from core client for 30 sec - exiting 02:30:56 (10232): No heartbeat from core client for 30 sec - exiting 02:30:57 (10232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:28:17 (3940): No heartbeat from core client for 30 sec - exiting 13:28:19 (3940): No heartbeat from core client for 30 sec - exiting 13:28:20 (3940): No heartbeat from core client for 30 sec - exiting 13:28:21 (3940): No heartbeat from core client for 30 sec - exiting 13:28:22 (3940): No heartbeat from core client for 30 sec - exiting 13:28:23 (3940): No heartbeat from core client for 30 sec - exiting 13:28:24 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:37:48 (3300): No heartbeat from core client for 30 sec - exiting 10:37:49 (3300): No heartbeat from core client for 30 sec - exiting 10:37:50 (3300): No heartbeat from core client for 30 sec - exiting 10:37:51 (3300): No heartbeat from core client for 30 sec - exiting 10:37:52 (3300): No heartbeat from core client for 30 sec - exiting 10:37:53 (3300): No heartbeat from core client for 30 sec - exiting 10:37:54 (3300): No heartbeat from core client for 30 sec - exiting 10:37:55 (3300): No heartbeat from core client for 30 sec - exiting 10:37:56 (3300): No heartbeat from core client for 30 sec - exiting 10:37:57 (3300): No heartbeat from core client for 30 sec - exiting 10:37:59 (3300): No heartbeat from core client for 30 sec - exiting 10:38:00 (3300): No heartbeat from core client for 30 sec - exiting 10:38:01 (3300): No heartbeat from core client for 30 sec - exiting 10:38:02 (3300): No heartbeat from core client for 30 sec - exiting 10:38:03 (3300): No heartbeat from core client for 30 sec - exiting 10:38:04 (3300): No heartbeat from core client for 30 sec - exiting 10:38:05 (3300): No heartbeat from core client for 30 sec - exiting 10:38:06 (3300): No heartbeat from core client for 30 sec - exiting 10:38:07 (3300): No heartbeat from core client for 30 sec - exiting 10:38:08 (3300): No heartbeat from core client for 30 sec - exiting 10:38:10 (3300): No heartbeat from core client for 30 sec - exiting 10:38:11 (3300): No heartbeat from core client for 30 sec - exiting 10:38:12 (3300): No heartbeat from core client for 30 sec - exiting 10:38:13 (3300): No heartbeat from core client for 30 sec - exiting 10:38:14 (3300): No heartbeat from core client for 30 sec - exiting 10:38:15 (3300): No heartbeat from core client for 30 sec - exiting 10:38:16 (3300): No heartbeat from core client for 30 sec - exiting 10:38:17 (3300): No heartbeat from core client for 30 sec - exiting 10:38:18 (3300): No heartbeat from core client for 30 sec - exiting 10:38:19 (3300): No heartbeat from core client for 30 sec - exiting 10:38:20 (3300): No heartbeat from core client for 30 sec - exiting 10:38:22 (3300): No heartbeat from core client for 30 sec - exiting 10:38:23 (3300): No heartbeat from core client for 30 sec - exiting 10:38:24 (3300): No heartbeat from core client for 30 sec - exiting 10:38:25 (3300): No heartbeat from core client for 30 sec - exiting 10:38:26 (3300): No heartbeat from core client for 30 sec - exiting 10:38:27 (3300): No heartbeat from core client for 30 sec - exiting 10:38:28 (3300): No heartbeat from core client for 30 sec - exiting 10:38:29 (3300): No heartbeat from core client for 30 sec - exiting 10:38:30 (3300): No heartbeat from core client for 30 sec - exiting 10:38:31 (3300): No heartbeat from core client for 30 sec - exiting 10:38:32 (3300): No heartbeat from core client for 30 sec - exiting 10:38:34 (3300): No heartbeat from core client for 30 sec - exiting 10:38:35 (3300): No heartbeat from core client for 30 sec - exiting 10:38:36 (3300): No heartbeat from core client for 30 sec - exiting 10:38:37 (3300): No heartbeat from core client for 30 sec - exiting 10:38:38 (3300): No heartbeat from core client for 30 sec - exiting 10:38:39 (3300): No heartbeat from core client for 30 sec - exiting 10:38:40 (3300): No heartbeat from core client for 30 sec - exiting 10:38:41 (3300): No heartbeat from core client for 30 sec - exiting 10:38:42 (3300): No heartbeat from core client for 30 sec - exiting 10:38:43 (3300): No heartbeat from core client for 30 sec - exiting 10:38:44 (3300): No heartbeat from core client for 30 sec - exiting 10:38:46 (3300): No heartbeat from core client for 30 sec - exiting 10:38:47 (3300): No heartbeat from core client for 30 sec - exiting 10:38:48 (3300): No heartbeat from core client for 30 sec - exiting 10:38:49 (3300): No heartbeat from core client for 30 sec - exiting 10:38:50 (3300): No heartbeat from core client for 30 sec - exiting 10:38:51 (3300): No heartbeat from core client for 30 sec - exiting 10:38:52 (3300): No heartbeat from core client for 30 sec - exiting 10:38:53 (3300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:54 (3300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:25:26 (1864): No heartbeat from core client for 30 sec - exiting 09:25:28 (1864): No heartbeat from core client for 30 sec - exiting 09:25:29 (1864): No heartbeat from core client for 30 sec - exiting 09:25:30 (1864): No heartbeat from core client for 30 sec - exiting 09:25:31 (1864): No heartbeat from core client for 30 sec - exiting 09:25:32 (1864): No heartbeat from core client for 30 sec - exiting 09:25:33 (1864): No heartbeat from core client for 30 sec - exiting 09:25:34 (1864): No heartbeat from core client for 30 sec - exiting 09:25:35 (1864): No heartbeat from core client for 30 sec - exiting 09:25:37 (1864): No heartbeat from core client for 30 sec - exiting 09:25:38 (1864): No heartbeat from core client for 30 sec - exiting 09:25:39 (1864): No heartbeat from core client for 30 sec - exiting 09:25:40 (1864): No heartbeat from core client for 30 sec - exiting 09:25:41 (1864): No heartbeat from core client for 30 sec - exiting 09:25:42 (1864): No heartbeat from core client for 30 sec - exiting 09:25:43 (1864): No heartbeat from core client for 30 sec - exiting 09:25:44 (1864): No heartbeat from core client for 30 sec - exiting 09:25:45 (1864): No heartbeat from core client for 30 sec - exiting 09:25:46 (1864): No heartbeat from core client for 30 sec - exiting 09:25:48 (1864): No heartbeat from core client for 30 sec - exiting 09:25:49 (1864): No heartbeat from core client for 30 sec - exiting 09:25:50 (1864): No heartbeat from core client for 30 sec - exiting 09:25:51 (1864): No heartbeat from core client for 30 sec - exiting 09:25:52 (1864): No heartbeat from core client for 30 sec - exiting 09:25:53 (1864): No heartbeat from core client for 30 sec - exiting 09:25:54 (1864): No heartbeat from core client for 30 sec - exiting 09:25:55 (1864): No heartbeat from core client for 30 sec - exiting 09:25:56 (1864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10016, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Jul 2011 09:37:17	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	207,360	378,797	1.8268
26 Jul 2011 20:27:10	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	181,440	334,503	1.8436
26 Jul 2011 05:19:51	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	155,520	287,728	1.8501
25 Jul 2011 18:18:09	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	129,600	240,201	1.8534
25 Jul 2011 17:49:56	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	103,680	191,855	1.8505
25 Jul 2011 16:23:55	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	77,760	143,695	1.8479
25 Jul 2011 15:56:10	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	51,840	96,115	1.8541
25 Jul 2011 15:34:58	1147730	13139792	hadcm3n_ykj8_1900_40_007359470_2	25,920	48,425	1.8682