Task 16005777

Name	hadcm3n_o5t4_1980_40_008387718_1
Workunit	8538577
Created	6 Sep 2013, 14:03:51 UTC
Sent	6 Sep 2013, 14:16:29 UTC
Report deadline	6 Dec 2013, 21:43:40 UTC
Received	19 Sep 2013, 9:04:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1207042
Run time	2 days 1 hours 46 min 31 sec
CPU time	1 days 13 hours 35 min 24 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:14:31 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:33 (5444): No heartbeat from core client for 30 sec - exiting 08:28:23 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:32 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:38 (4712): No heartbeat from core client for 30 sec - exiting 08:34:39 (4712): No heartbeat from core client for 30 sec - exiting 08:34:43 (4712):09:34:41 (2212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:42 (2212): No heartbeat from core client for 30 sec - exiting 09:39:43 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:39:47 (5312): No heartbeat from core client for 30 sec - exiting 09:39:50 (5312): No heartbeat from core client for 30 sec - exiting 09:39:51 (5312): No heartbeat from core client for 30 sec - exiting 09:39:52 (5312): No heartbeat from core client for 30 sec - exiting 09:39:53 (5312): No heartbeat from core client for 30 sec - exiting 09:39:54 (5312): No heartbeat from core client for 30 sec - exiting 09:39:55 (5312): No heartbeat from core client for 30 sec - exiting 09:39:56 (5312): No heartbeat from core client for 30 sec - exiting 09:39:57 (5312): No heartbeat from core client for 30 sec - exiting 09:39:58 (5312): No heartbeat from core client for 30 sec - exiting 09:39:59 (5312): No heartbeat from core client for 30 sec - exiting 09:40:00 (5312): No heartbeat from core client for 30 sec - exiting 09:40:01 (5312): No heartbeat from core client for 30 sec - exiting 09:40:03 (5312): No heartbeat from core client for 30 sec - exiting 09:40:04 (5312): No heartbeat from core client for 30 sec - exiting 09:40:05 (5312): No heartbeat from core client for 30 sec - exiting 09:47:41 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:49:20 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:05 (3232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:08 (3232): No heartbeat from core client for 30 sec - exiting 09:55:09 (3232): No heartbeat from core client for 30 sec - exiting 10:00:05 (3728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:07 (3728): No heartbeat from core client for 30 sec - exiting 10:00:08 (3728): No heartbeat from core client for 30 sec - exiting 10:00:09 (3728): No heartbeat from core client for 30 sec - exiting 10:00:10 (3728): No heartbeat from core client for 30 sec - exiting 10:00:11 (3728): No heartbeat from core client for 30 sec - exiting 10:00:12 (3728): No heartbeat from core client for 30 sec - exiting 10:00:13 (3728): No heartbeat from core client for 30 sec - exiting 10:00:14 (3728): No heartbeat from core client for 30 sec - exiting 10:05:05 (5008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:12 (5008): No heartbeat from core client for 30 sec - exiting 10:05:14 (5008): No heartbeat from core client for 30 sec - exiting 10:05:16 (5008): No heartbeat from core client for 30 sec - exiting 10:05:20 (5008): No heartbeat from core client for 30 sec - exiting 10:13:41 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:13:47 (5916): No heartbeat from core client for 30 sec - exiting 10:23:25 (2680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:37:49 (404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:48 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:50 (4664): No heartbeat from core client for 30 sec - exiting 11:45:51 (4664): No heartbeat from core client for 30 sec - exiting 11:47:04 (8004): No heartbeat from core client for 30 sec - exiting 11:47:05 (8004): No heartbeat from core client for 30 sec - exiting 11:47:06 (8004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:32:16 (1524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:32:17 (1524): No heartbeat from core client for 30 sec - exiting 15:38:45 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:50 (5268): No heartbeat from core client for 30 sec - exiting 15:38:53 (5268): No heartbeat from core client for 30 sec - exiting 15:47:14 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:47:18 (5532): No heartbeat from core client for 30 sec - exiting 15:47:19 (5532): No heartbeat from core client for 30 sec - exiting 15:47:20 (5532): No heartbeat from core client for 30 sec - exiting 15:47:22 (5532): No heartbeat from core client for 30 sec - exiting 15:47:23 (5532): No heartbeat from core client for 30 sec - exiting 15:47:24 (5532): No heartbeat from core client for 30 sec - exiting 15:47:25 (5532): No heartbeat from core client for 30 sec - exiting 15:47:26 (5532): No heartbeat from core client for 30 sec - exiting 15:47:27 (5532): No heartbeat from core client for 30 sec - exiting 15:47:28 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:14:29 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:30 (4408): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:24:36 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:24:38 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:21:47 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:22:09 (2776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:27 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:56:21 (1076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:52:39 (2912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 08:05:47 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Sep 2013 02:34:12	1207042	16005777	hadcm3n_o5t4_1980_40_008387718_1	77,760	116,059	1.4925
14 Sep 2013 08:27:52	1207042	16005777	hadcm3n_o5t4_1980_40_008387718_1	51,840	79,206	1.5279
11 Sep 2013 09:06:15	1207042	16005777	hadcm3n_o5t4_1980_40_008387718_1	25,920	41,557	1.6033