Task 13095266

Name	hadcm3n_y9wq_1900_40_007345700_1
Workunit	7543130
Created	6 Jul 2011, 13:32:55 UTC
Sent	20 Jul 2011, 4:59:49 UTC
Report deadline	19 Oct 2011, 12:27:00 UTC
Received	8 Aug 2011, 17:54:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1197373
Run time	10 days 18 hours 42 min 23 sec
CPU time	10 days 4 hours 57 min 23 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	1.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 22:42:31 (16071): No heartbeat from core client for 30 sec - exiting 22:44:28 (16071): No heartbeat from core client for 30 sec - exiting 22:45:05 (16071): No heartbeat from core client for 30 sec - exiting 22:45:41 (16071): No heartbeat from core client for 30 sec - exiting 22:45:42 (16071): No heartbeat from core client for 30 sec - exiting 22:45:43 (16071): No heartbeat from core client for 30 sec - exiting 22:45:44 (16071): No heartbeat from core client for 30 sec - exiting 22:45:45 (16071): No heartbeat from core client for 30 sec - exiting 22:45:46 (16071): No heartbeat from core client for 30 sec - exiting 22:45:47 (16071): No heartbeat from core client for 30 sec - exiting 22:45:48 (16071): No heartbeat from core client for 30 sec - exiting 22:45:49 (16071): No heartbeat from core client for 30 sec - exiting 22:45:50 (16071): No heartbeat from core client for 30 sec - exiting 22:45:51 (16071): No heartbeat from core client for 30 sec - exiting 22:45:52 (16071): No heartbeat from core client for 30 sec - exiting 22:45:53 (16071): No heartbeat from core client for 30 sec - exiting 22:45:54 (16071): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 22:46:50 (16174): No heartbeat from core client for 30 sec - exiting 22:46:51 (16174): No heartbeat from core client for 30 sec - exiting 22:46:52 (16174): No heartbeat from core client for 30 sec - exiting 22:46:53 (16174): No heartbeat from core client for 30 sec - exiting 22:46:54 (16174): No heartbeat from core client for 30 sec - exiting 22:46:55 (16174): No heartbeat from core client for 30 sec - exiting 22:46:56 (16174): No heartbeat from core client for 30 sec - exiting 22:46:57 (16174): No heartbeat from core client for 30 sec - exiting 22:46:58 (16174): No heartbeat from core client for 30 sec - exiting 22:46:59 (16174): No heartbeat from core client for 30 sec - exiting 22:47:00 (16174): No heartbeat from core client for 30 sec - exiting 22:47:01 (16174): No heartbeat from core client for 30 sec - exiting 22:47:02 (16174): No heartbeat from core client for 30 sec - exiting 22:47:03 (16174): No heartbeat from core client for 30 sec - exiting 22:47:04 (16174): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 22:48:01 (16206): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day 01:30:25 (1375): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 01:33:44 (1385): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day 01:39:33 (1394): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 01:41:48 (1403): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day 01:45:47 (1403): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 01:48:05 (1410): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/ocean_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 02:00:54 (1422): No heartbeat from core client for 30 sec - exiting 02:00:55 (1422): No heartbeat from core client for 30 sec - exiting 02:00:56 (1422): No heartbeat from core client for 30 sec - exiting 02:00:57 (1422): No heartbeat from core client for 30 sec - exiting 02:00:58 (1422): No heartbeat from core client for 30 sec - exiting 02:00:59 (1422): No heartbeat from core client for 30 sec - exiting 02:01:00 (1422): No heartbeat from core client for 30 sec - exiting 02:01:01 (1422): No heartbeat from core client for 30 sec - exiting 02:01:02 (1422): No heartbeat from core client for 30 sec - exiting 02:01:03 (1422): No heartbeat from core client for 30 sec - exiting 02:01:04 (1422): No heartbeat from core client for 30 sec - exiting 02:01:05 (1422): No heartbeat from core client for 30 sec - exiting 02:01:06 (1422): No heartbeat from core client for 30 sec - exiting 02:01:07 (1422): No heartbeat from core client for 30 sec - exiting 02:01:08 (1422): No heartbeat from core client for 30 sec - exiting 02:01:09 (1422): No heartbeat from core client for 30 sec - exiting 02:01:10 (1422): No heartbeat from core client for 30 sec - exiting 02:01:11 (1422): No heartbeat from core client for 30 sec - exiting 02:01:12 (1422): No heartbeat from core client for 30 sec - exiting 02:01:13 (1422): No heartbeat from core client for 30 sec - exiting 02:01:14 (1422): No heartbeat from core client for 30 sec - exiting 02:01:15 (1422): No heartbeat from core client for 30 sec - exiting 02:01:16 (1422): No heartbeat from core client for 30 sec - exiting 02:01:17 (1422): No heartbeat from core client for 30 sec - exiting 02:01:18 (1422): No heartbeat from core client for 30 sec - exiting 02:01:19 (1422): No heartbeat from core client for 30 sec - exiting 02:01:20 (1422): No heartbeat from core client for 30 sec - exiting 02:01:21 (1422): No heartbeat from core client for 30 sec - exiting 02:01:22 (1422): No heartbeat from core client for 30 sec - exiting 02:01:23 (1422): No heartbeat from core client for 30 sec - exiting 02:01:24 (1422): No heartbeat from core client for 30 sec - exiting 02:01:25 (1422): No heartbeat from core client for 30 sec - exiting 02:01:26 (1422): No heartbeat from core client for 30 sec - exiting 02:01:27 (1422): No heartbeat from core client for 30 sec - exiting 02:01:28 (1422): No heartbeat from core client for 30 sec - exiting 02:01:29 (1422): No heartbeat from core client for 30 sec - exiting 02:01:30 (1422): No heartbeat from core client for 30 sec - exiting 02:01:31 (1422): No heartbeat from core client for 30 sec - exiting 02:01:32 (1422): No heartbeat from core client for 30 sec - exiting 02:01:33 (1422): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 02:07:59 (1439): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:40 (1615): No heartbeat from core client for 30 sec - exiting 02:09:41 (1615): No heartbeat from core client for 30 sec - exiting 02:09:42 (1615): No heartbeat from core client for 30 sec - exiting 02:09:43 (1615): No heartbeat from core client for 30 sec - exiting 02:09:44 (1615): No heartbeat from core client for 30 sec - exiting 02:09:45 (1615): No heartbeat from core client for 30 sec - exiting 02:09:46 (1615): No heartbeat from core client for 30 sec - exiting 02:09:47 (1615): No heartbeat from core client for 30 sec - exiting 02:09:48 (1615): No heartbeat from core client for 30 sec - exiting 02:09:49 (1615): No heartbeat from core client for 30 sec - exiting 02:09:50 (1615): No heartbeat from core client for 30 sec - exiting 02:09:51 (1615): No heartbeat from core client for 30 sec - exiting 02:09:52 (1615): No heartbeat from core client for 30 sec - exiting 02:09:53 (1615): No heartbeat from core client for 30 sec - exiting 02:09:54 (1615): No heartbeat from core client for 30 sec - exiting 02:09:55 (1615): No heartbeat from core client for 30 sec - exiting 02:10:27 (1615): No heartbeat from core client for 30 sec - exiting 02:10:28 (1615): No heartbeat from core client for 30 sec - exiting 02:10:29 (1615): No heartbeat from core client for 30 sec - exiting 02:10:30 (1615): No heartbeat from core client for 30 sec - exiting 02:10:31 (1615): No heartbeat from core client for 30 sec - exiting 02:10:32 (1615): No heartbeat from core client for 30 sec - exiting 02:10:33 (1615): No heartbeat from core client for 30 sec - exiting 02:10:34 (1615): No heartbeat from core client for 30 sec - exiting 02:10:35 (1615): No heartbeat from core client for 30 sec - exiting 02:10:36 (1615): No heartbeat from core client for 30 sec - exiting 02:10:37 (1615): No heartbeat from core client for 30 sec - exiting 02:10:38 (1615): No heartbeat from core client for 30 sec - exiting 02:10:39 (1615): No heartbeat from core client for 30 sec - exiting 02:10:40 (1615): No heartbeat from core client for 30 sec - exiting 02:10:41 (1615): No heartbeat from core client for 30 sec - exiting 02:10:42 (1615): No heartbeat from core client for 30 sec - exiting 02:10:43 (1615): No heartbeat from core client for 30 sec - exiting 02:10:44 (1615): No heartbeat from core client for 30 sec - exiting 02:10:45 (1615): No heartbeat from core client for 30 sec - exiting 02:10:46 (1615): No heartbeat from core client for 30 sec - exiting 02:10:47 (1615): No heartbeat from core client for 30 sec - exiting 02:10:48 (1615): No heartbeat from core client for 30 sec - exiting 02:10:49 (1615): No heartbeat from core client for 30 sec - exiting 02:10:50 (1615): No heartbeat from core client for 30 sec - exiting 02:10:51 (1615): No heartbeat from core client for 30 sec - exiting 02:10:52 (1615): No heartbeat from core client for 30 sec - exiting 02:10:53 (1615): No heartbeat from core client for 30 sec - exiting 02:10:54 (1615): No heartbeat from core client for 30 sec - exiting 02:10:55 (1615): No heartbeat from core client for 30 sec - exiting 02:10:56 (1615): No heartbeat from core client for 30 sec - exiting 02:10:57 (1615): No heartbeat from core client for 30 sec - exiting 02:10:58 (1615): No heartbeat from core client for 30 sec - exiting 02:10:59 (1615): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 02:12:49 (1640): No heartbeat from core client for 30 sec - exiting 02:13:33 (1640): No heartbeat from core client for 30 sec - exiting 02:13:34 (1640): No heartbeat from core client for 30 sec - exiting 02:13:35 (1640): No heartbeat from core client for 30 sec - exiting 02:13:36 (1640): No heartbeat from core client for 30 sec - exiting 02:13:37 (1640): No heartbeat from core client for 30 sec - exiting 02:13:38 (1640): No heartbeat from core client for 30 sec - exiting 02:13:39 (1640): No heartbeat from core client for 30 sec - exiting 02:13:40 (1640): No heartbeat from core client for 30 sec - exiting 02:13:41 (1640): No heartbeat from core client for 30 sec - exiting 02:13:42 (1640): No heartbeat from core client for 30 sec - exiting 02:13:43 (1640): No heartbeat from core client for 30 sec - exiting 02:13:44 (1640): No heartbeat from core client for 30 sec - exiting 02:13:45 (1640): No heartbeat from core client for 30 sec - exiting 02:13:46 (1640): No heartbeat from core client for 30 sec - exiting 02:13:47 (1640): No heartbeat from core client for 30 sec - exiting 02:13:48 (1640): No heartbeat from core client for 30 sec - exiting 02:13:49 (1640): No heartbeat from core client for 30 sec - exiting 02:13:50 (1640): No heartbeat from core client for 30 sec - exiting 02:13:51 (1640): No heartbeat from core client for 30 sec - exiting 02:13:52 (1640): No heartbeat from core client for 30 sec - exiting 02:13:53 (1640): No heartbeat from core client for 30 sec - exiting 02:13:54 (1640): No heartbeat from core client for 30 sec - exiting 02:13:55 (1640): No heartbeat from core client for 30 sec - exiting 02:13:56 (1640): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 02:17:44 (1735): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 02:20:15 (1759): No heartbeat from core client for 30 sec - exiting 02:20:17 (1759): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 02:28:38 (1767): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:16 (1788): No heartbeat from core client for 30 sec - exiting 02:31:17 (1788): No heartbeat from core client for 30 sec - exiting 02:31:18 (1788): No heartbeat from core client for 30 sec - exiting 02:31:19 (1788): No heartbeat from core client for 30 sec - exiting 02:31:20 (1788): No heartbeat from core client for 30 sec - exiting 02:31:21 (1788): No heartbeat from core client for 30 sec - exiting 02:31:22 (1788): No heartbeat from core client for 30 sec - exiting 02:31:23 (1788): No heartbeat from core client for 30 sec - exiting 02:31:24 (1788): No heartbeat from core client for 30 sec - exiting 02:31:25 (1788): No heartbeat from core client for 30 sec - exiting 02:31:26 (1788): No heartbeat from core client for 30 sec - exiting 02:31:27 (1788): No heartbeat from core client for 30 sec - exiting 02:31:28 (1788): No heartbeat from core client for 30 sec - exiting 02:31:29 (1788): No heartbeat from core client for 30 sec - exiting 02:31:30 (1788): No heartbeat from core client for 30 sec - exiting 02:31:31 (1788): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day 02:39:30 (1798): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 CPDN Monitor - No 'heartbeat' from BOINC... 02:40:13 (1808): No heartbeat from core client for 30 sec - exiting 02:41:23 (1808): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... 02:44:32 (1827): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day 02:45:19 (1827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:27 (1833): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_y9wq_1900_40_007345700/dataout/atmos_restart.day BUFFIN: Read Failed: Input/output error BUFFIN: C I/O Error ferror - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Aug 2011 19:19:36	1158954	13095266	hadcm3n_y9wq_1900_40_007345700_1	155,520	855,220	5.4991
04 Aug 2011 03:51:17	1158954	13095266	hadcm3n_y9wq_1900_40_007345700_1	129,600	712,910	5.5008
02 Aug 2011 10:19:26	1158954	13095266	hadcm3n_y9wq_1900_40_007345700_1	103,680	570,793	5.5053
31 Jul 2011 17:38:46	1158954	13095266	hadcm3n_y9wq_1900_40_007345700_1	77,760	428,569	5.5114
29 Jul 2011 22:45:14	1158954	13095266	hadcm3n_y9wq_1900_40_007345700_1	51,840	285,833	5.5138
25 Jul 2011 19:28:07	1158954	13095266	hadcm3n_y9wq_1900_40_007345700_1	25,920	143,802	5.5479