Task 13615988

Name	hadcm3n_o7l6_1980_40_007540342_4
Workunit	7737574
Created	7 Nov 2011, 14:02:55 UTC
Sent	7 Nov 2011, 14:06:21 UTC
Report deadline	6 Feb 2012, 21:33:32 UTC
Received	18 Dec 2011, 12:59:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1170513
Run time	2 days 5 hours 59 min 46 sec
CPU time	1 days 8 hours 45 min 24 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.2</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 01:48:59 (171172): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 16:11:58 (35784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:12:00 (35784): No heartbeat from core client for 30 sec - exiting 16:12:01 (35784): No heartbeat from core client for 30 sec - exiting 16:12:02 (35784): No heartbeat from core client for 30 sec - exiting 16:12:03 (35784): No heartbeat from core client for 30 sec - exiting 16:12:04 (35784): No heartbeat from core client for 30 sec - exiting 16:12:11 (35784): No heartbeat from core client for 30 sec - exiting 16:12:12 (35784): No heartbeat from core client for 30 sec - exiting 16:12:13 (35784): No heartbeat from core client for 30 sec - exiting 16:12:14 (35784): No heartbeat from core client for 30 sec - exiting 16:12:15 (35784): No heartbeat from core client for 30 sec - exiting 16:12:16 (35784): No heartbeat from core client for 30 sec - exiting 16:12:17 (35784): No heartbeat from core client for 30 sec - exiting 16:12:18 (35784): No heartbeat from core client for 30 sec - exiting 16:12:19 (35784): No heartbeat from core client for 30 sec - exiting 16:12:20 (35784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:40:01 (34580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:03 (34580): No heartbeat from core client for 30 sec - exiting 15:40:04 (34580): No heartbeat from core client for 30 sec - exiting 15:40:05 (34580): No heartbeat from core client for 30 sec - exiting 15:48:28 (33580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:53:02 (38904): No heartbeat from core client for 30 sec - exiting 15:53:03 (38904): No heartbeat from core client for 30 sec - exiting 15:53:04 (38904): No heartbeat from core client for 30 sec - exiting 15:53:05 (38904): No heartbeat from core client for 30 sec - exiting 15:53:06 (38904): No heartbeat from core client for 30 sec - exiting 15:53:07 (38904): No heartbeat from core client for 30 sec - exiting 15:53:08 (38904): No heartbeat from core client for 30 sec - exiting 15:53:09 (38904): No heartbeat from core client for 30 sec - exiting 15:53:10 (38904): No heartbeat from core client for 30 sec - exiting 15:53:11 (38904): No heartbeat from core client for 30 sec - exiting 15:53:12 (38904): No heartbeat from core client for 30 sec - exiting 15:53:13 (38904): No heartbeat from core client for 30 sec - exiting 15:53:14 (38904): No heartbeat from core client for 30 sec - exiting 15:53:15 (38904): No heartbeat from core client for 30 sec - exiting 15:53:16 (38904): No heartbeat from core client for 30 sec - exiting 15:53:17 (38904): No heartbeat from core client for 30 sec - exiting 15:53:18 (38904): No heartbeat from core client for 30 sec - exiting 15:53:19 (38904): No heartbeat from core client for 30 sec - exiting 15:53:20 (38904): No heartbeat from core client for 30 sec - exiting 15:53:21 (38904): No heartbeat from core client for 30 sec - exiting 15:53:22 (38904): No heartbeat from core client for 30 sec - exiting 15:53:23 (38904): No heartbeat from core client for 30 sec - exiting 15:53:24 (38904): No heartbeat from core client for 30 sec - exiting 15:53:25 (38904): No heartbeat from core client for 30 sec - exiting 15:53:26 (38904): No heartbeat from core client for 30 sec - exiting 15:53:27 (38904): No heartbeat from core client for 30 sec - exiting 15:53:28 (38904): No heartbeat from core client for 30 sec - exiting 15:53:29 (38904): No heartbeat from core client for 30 sec - exiting 15:53:30 (38904): No heartbeat from core client for 30 sec - exiting 15:53:31 (38904): No heartbeat from core client for 30 sec - exiting 15:53:32 (38904): No heartbeat from core client for 30 sec - exiting 15:53:33 (38904): No heartbeat from core client for 30 sec - exiting 15:53:34 (38904): No heartbeat from core client for 30 sec - exiting 15:53:35 (38904): No heartbeat from core client for 30 sec - exiting 15:53:36 (38904): No heartbeat from core client for 30 sec - exiting 15:53:37 (38904): No heartbeat from core client for 30 sec - exiting 15:53:38 (38904): No heartbeat from core client for 30 sec - exiting 15:53:39 (38904): No heartbeat from core client for 30 sec - exiting 15:53:40 (38904): No heartbeat from core client for 30 sec - exiting 15:53:41 (38904): No heartbeat from core client for 30 sec - exiting 15:53:42 (38904): No heartbeat from core client for 30 sec - exiting 15:53:43 (38904): No heartbeat from core client for 30 sec - exiting 15:53:44 (38904): No heartbeat from core client for 30 sec - exiting 15:53:45 (38904): No heartbeat from core client for 30 sec - exiting 15:53:46 (38904): No heartbeat from core client for 30 sec - exiting 15:53:47 (38904): No heartbeat from core client for 30 sec - exiting 15:53:48 (38904): No heartbeat from core client for 30 sec - exiting 15:53:49 (38904): No heartbeat from core client for 30 sec - exiting 15:53:50 (38904): No heartbeat from core client for 30 sec - exiting 15:53:51 (38904): No heartbeat from core client for 30 sec - exiting 15:53:52 (38904): No heartbeat from core client for 30 sec - exiting 15:53:53 (38904): No heartbeat from core client for 30 sec - exiting 15:53:54 (38904): No heartbeat from core client for 30 sec - exiting 15:53:55 (38904): No heartbeat from core client for 30 sec - exiting 15:53:56 (38904): No heartbeat from core client for 30 sec - exiting 15:53:57 (38904): No heartbeat from core client for 30 sec - exiting 15:53:58 (38904): No heartbeat from core client for 30 sec - exiting 15:53:59 (38904): No heartbeat from core client for 30 sec - exiting 15:54:00 (38904): No heartbeat from core client for 30 sec - exiting 15:54:01 (38904): No heartbeat from core client for 30 sec - exiting 15:54:02 (38904): No heartbeat from core client for 30 sec - exiting 15:54:03 (38904): No heartbeat from core client for 30 sec - exiting 15:54:04 (38904): No heartbeat from core client for 30 sec - exiting 15:54:05 (38904): No heartbeat from core client for 30 sec - exiting 15:54:06 (38904): No heartbeat from core client for 30 sec - exiting 15:54:07 (38904): No heartbeat from core client for 30 sec - exiting 15:54:08 (38904): No heartbeat from core client for 30 sec - exiting 15:54:09 (38904): No heartbeat from core client for 30 sec - exiting 15:54:10 (38904): No heartbeat from core client for 30 sec - exiting 15:54:11 (38904): No heartbeat from core client for 30 sec - exiting 15:54:12 (38904): No heartbeat from core client for 30 sec - exiting 15:54:13 (38904): No heartbeat from core client for 30 sec - exiting 15:54:14 (38904): No heartbeat from core client for 30 sec - exiting 15:54:15 (38904): No heartbeat from core client for 30 sec - exiting 15:54:16 (38904): No heartbeat from core client for 30 sec - exiting 15:54:17 (38904): No heartbeat from core client for 30 sec - exiting 15:54:18 (38904): No heartbeat from core client for 30 sec - exiting 15:54:19 (38904): No heartbeat from core client for 30 sec - exiting 15:54:20 (38904): No heartbeat from core client for 30 sec - exiting 15:54:21 (38904): No heartbeat from core client for 30 sec - exiting 15:54:22 (38904): No heartbeat from core client for 30 sec - exiting 15:54:23 (38904): No heartbeat from core client for 30 sec - exiting 15:54:24 (38904): No heartbeat from core client for 30 sec - exiting 15:54:25 (38904): No heartbeat from core client for 30 sec - exiting 15:54:26 (38904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:18:45 (35984): No heartbeat from core client for 30 sec - exiting 17:18:46 (35984): No heartbeat from core client for 30 sec - exiting 17:18:47 (35984): No heartbeat from core client for 30 sec - exiting 17:18:48 (35984): No heartbeat from core client for 30 sec - exiting 17:18:49 (35984): No heartbeat from core client for 30 sec - exiting 17:18:50 (35984): No heartbeat from core client for 30 sec - exiting 17:18:51 (35984): No heartbeat from core client for 30 sec - exiting 17:18:52 (35984): No heartbeat from core client for 30 sec - exiting 17:18:53 (35984): No heartbeat from core client for 30 sec - exiting 17:18:54 (35984): No heartbeat from core client for 30 sec - exiting 17:18:55 (35984): No heartbeat from core client for 30 sec - exiting 17:18:56 (35984): No heartbeat from core client for 30 sec - exiting 17:18:57 (35984): No heartbeat from core client for 30 sec - exiting 17:18:58 (35984): No heartbeat from core client for 30 sec - exiting 17:18:59 (35984): No heartbeat from core client for 30 sec - exiting 17:19:00 (35984): No heartbeat from core client for 30 sec - exiting 17:19:01 (35984): No heartbeat from core client for 30 sec - exiting 17:19:02 (35984): No heartbeat from core client for 30 sec - exiting 17:19:03 (35984): No heartbeat from core client for 30 sec - exiting 17:19:04 (35984): No heartbeat from core client for 30 sec - exiting 17:19:05 (35984): No heartbeat from core client for 30 sec - exiting 17:19:06 (35984): No heartbeat from core client for 30 sec - exiting 17:19:07 (35984): No heartbeat from core client for 30 sec - exiting 17:19:08 (35984): No heartbeat from core client for 30 sec - exiting 17:19:09 (35984): No heartbeat from core client for 30 sec - exiting 17:19:10 (35984): No heartbeat from core client for 30 sec - exiting 17:19:11 (35984): No heartbeat from core client for 30 sec - exiting 17:19:12 (35984): No heartbeat from core client for 30 sec - exiting 17:19:13 (35984): No heartbeat from core client for 30 sec - exiting 17:19:14 (35984): No heartbeat from core client for 30 sec - exiting 17:19:15 (35984): No heartbeat from core client for 30 sec - exiting 17:19:16 (35984): No heartbeat from core client for 30 sec - exiting 17:19:17 (35984): No heartbeat from core client for 30 sec - exiting 17:19:18 (35984): No heartbeat from core client for 30 sec - exiting 17:19:19 (35984): No heartbeat from core client for 30 sec - exiting 17:19:20 (35984): No heartbeat from core client for 30 sec - exiting 17:19:21 (35984): No heartbeat from core client for 30 sec - exiting 17:19:22 (35984): No heartbeat from core client for 30 sec - exiting 17:19:23 (35984): No heartbeat from core client for 30 sec - exiting 17:19:24 (35984): No heartbeat from core client for 30 sec - exiting 17:19:25 (35984): No heartbeat from core client for 30 sec - exiting 17:19:26 (35984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:44:07 (32824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:53:26 (41440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:53:29 (41440): No heartbeat from core client for 30 sec - exiting 15:53:30 (41440): No heartbeat from core client for 30 sec - exiting 15:53:31 (41440): No heartbeat from core client for 30 sec - exiting 15:53:32 (41440): No heartbeat from core client for 30 sec - exiting 15:53:33 (41440): No heartbeat from core client for 30 sec - exiting 15:53:34 (41440): No heartbeat from core client for 30 sec - exiting 15:53:35 (41440): No heartbeat from core client for 30 sec - exiting 15:53:36 (41440): No heartbeat from core client for 30 sec - exiting 15:53:37 (41440): No heartbeat from core client for 30 sec - exiting 15:53:38 (41440): No heartbeat from core client for 30 sec - exiting 15:57:57 (17092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:38:32 (44932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:44:12 (44532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:45:25 (12280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:38 (45080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:39 (45080): No heartbeat from core client for 30 sec - exiting 16:49:40 (45080): No heartbeat from core client for 30 sec - exiting 16:49:41 (45080): No heartbeat from core client for 30 sec - exiting 16:49:42 (45080): No heartbeat from core client for 30 sec - exiting 16:49:43 (45080): No heartbeat from core client for 30 sec - exiting 16:49:44 (45080): No heartbeat from core client for 30 sec - exiting 16:49:45 (45080): No heartbeat from core client for 30 sec - exiting 16:49:46 (45080): No heartbeat from core client for 30 sec - exiting 16:51:50 (43212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:19 (36220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:12 (20600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:47 (43272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:51:26 (46320): No heartbeat from core client for 30 sec - exiting 17:51:27 (46320): No heartbeat from core client for 30 sec - exiting 17:51:28 (46320): No heartbeat from core client for 30 sec - exiting 17:51:29 (46320): No heartbeat from core client for 30 sec - exiting 17:51:30 (46320): No heartbeat from core client for 30 sec - exiting 17:51:31 (46320): No heartbeat from core client for 30 sec - exiting 17:51:32 (46320): No heartbeat from core client for 30 sec - exiting 17:51:33 (46320): No heartbeat from core client for 30 sec - exiting 17:51:34 (46320): No heartbeat from core client for 30 sec - exiting 17:51:35 (46320): No heartbeat from core client for 30 sec - exiting 17:51:36 (46320): No heartbeat from core client for 30 sec - exiting 17:51:37 (46320): No heartbeat from core client for 30 sec - exiting 17:51:38 (46320): No heartbeat from core client for 30 sec - exiting 17:51:39 (46320): No heartbeat from core client for 30 sec - exiting 17:51:40 (46320): No heartbeat from core client for 30 sec - exiting 17:51:41 (46320): No heartbeat from core client for 30 sec - exiting 17:51:43 (46320): No heartbeat from core client for 30 sec - exiting 17:51:44 (46320): No heartbeat from core client for 30 sec - exiting 17:51:45 (46320): No heartbeat from core client for 30 sec - exiting 17:51:46 (46320): No heartbeat from core client for 30 sec - exiting 17:51:47 (46320): No heartbeat from core client for 30 sec - exiting 17:51:48 (46320): No heartbeat from core client for 30 sec - exiting 17:51:49 (46320): No heartbeat from core client for 30 sec - exiting 17:51:50 (46320): No heartbeat from core client for 30 sec - exiting 17:51:51 (46320): No heartbeat from core client for 30 sec - exiting 17:51:52 (46320): No heartbeat from core client for 30 sec - exiting 17:51:53 (46320): No heartbeat from core client for 30 sec - exiting 17:51:54 (46320): No heartbeat from core client for 30 sec - exiting 17:51:55 (46320): No heartbeat from core client for 30 sec - exiting 17:51:56 (46320): No heartbeat from core client for 30 sec - exiting 17:51:57 (46320): No heartbeat from core client for 30 sec - exiting 17:51:58 (46320): No heartbeat from core client for 30 sec - exiting 17:51:59 (46320): No heartbeat from core client for 30 sec - exiting 17:52:00 (46320): No heartbeat from core client for 30 sec - exiting 17:52:01 (46320): No heartbeat from core client for 30 sec - exiting 17:52:02 (46320): No heartbeat from core client for 30 sec - exiting 17:52:03 (46320): No heartbeat from core client for 30 sec - exiting 17:52:04 (46320): No heartbeat from core client for 30 sec - exiting 17:52:05 (46320): No heartbeat from core client for 30 sec - exiting 17:52:06 (46320): No heartbeat from core client for 30 sec - exiting 17:52:07 (46320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:56 (46188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:58 (46188): No heartbeat from core client for 30 sec - exiting 19:34:00 (46188): No heartbeat from core client for 30 sec - exiting 19:34:01 (46188): No heartbeat from core client for 30 sec - exiting 19:34:02 (46188): No heartbeat from core client for 30 sec - exiting 19:34:03 (46188): No heartbeat from core client for 30 sec - exiting 19:34:04 (46188): No heartbeat from core client for 30 sec - exiting 19:34:05 (46188): No heartbeat from core client for 30 sec - exiting 19:34:06 (46188): No heartbeat from core client for 30 sec - exiting 19:34:07 (46188): No heartbeat from core client for 30 sec - exiting 19:34:08 (46188): No heartbeat from core client for 30 sec - exiting 19:40:18 (45912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: Access is denied. 19:41:33 (46476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:28 (20940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:18:34 (24264): start_timer_thread(): CreateThread() failed, errno 0 09:47:43 (23228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:44 (23228): No heartbeat from core client for 30 sec - exiting 09:47:45 (23228): No heartbeat from core client for 30 sec - exiting 09:47:46 (23228): No heartbeat from core client for 30 sec - exiting 09:47:47 (23228): No heartbeat from core client for 30 sec - exiting 09:47:48 (23228): No heartbeat from core client for 30 sec - exiting 09:47:49 (23228): No heartbeat from core client for 30 sec - exiting 09:47:50 (23228): No heartbeat from core client for 30 sec - exiting 09:47:51 (23228): No heartbeat from core client for 30 sec - exiting 09:47:52 (23228): No heartbeat from core client for 30 sec - exiting 09:47:53 (23228): No heartbeat from core client for 30 sec - exiting 09:47:54 (23228): No heartbeat from core client for 30 sec - exiting 09:47:55 (23228): No heartbeat from core client for 30 sec - exiting 09:47:56 (23228): No heartbeat from core client for 30 sec - exiting 09:48:51 (25328): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4036, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4036, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4036, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4036, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Dec 2011 04:28:11	1170513	13615988	hadcm3n_o7l6_1980_40_007540342_4	51,840	131,183	2.5305
26 Nov 2011 03:43:23	1170513	13615988	hadcm3n_o7l6_1980_40_007540342_4	25,920	68,057	2.6257