Task 15836087

Name	hadcm3n_o723_1980_40_008385985_1
Workunit	8536844
Created	9 Jun 2013, 4:24:15 UTC
Sent	9 Jun 2013, 4:36:00 UTC
Report deadline	8 Sep 2013, 12:03:11 UTC
Received	11 Jun 2013, 1:44:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1282401
Run time	1 days 14 hours 36 min 3 sec
CPU time	1 days 11 hours 46 min 42 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 13:50:51 (29691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:21 (34205): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:40 (34363): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:48 (34363): No heartbeat from core client for 30 sec - exiting 15:36:04 (34532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:05 (34532): No heartbeat from core client for 30 sec - exiting 15:36:06 (34532): No heartbeat from core client for 30 sec - exiting 15:36:07 (34532): No heartbeat from core client for 30 sec - exiting 15:40:17 (35446): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:18 (35446): No heartbeat from core client for 30 sec - exiting 15:40:19 (35446): No heartbeat from core client for 30 sec - exiting 15:40:20 (35446): No heartbeat from core client for 30 sec - exiting 15:40:21 (35446): No heartbeat from core client for 30 sec - exiting 15:40:22 (35446): No heartbeat from core client for 30 sec - exiting 15:40:23 (35446): No heartbeat from core client for 30 sec - exiting 15:40:24 (35446): No heartbeat from core client for 30 sec - exiting 15:40:25 (35446): No heartbeat from core client for 30 sec - exiting 15:40:26 (35446): No heartbeat from core client for 30 sec - exiting 15:40:27 (35446): No heartbeat from core client for 30 sec - exiting 15:40:28 (35446): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 15:49:04 (35605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:49:05 (35605): No heartbeat from core client for 30 sec - exiting 15:53:15 (35791): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:57:07 (35927): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:57:08 (35927): No heartbeat from core client for 30 sec - exiting 15:57:09 (35927): No heartbeat from core client for 30 sec - exiting 15:57:10 (35927): No heartbeat from core client for 30 sec - exiting 15:57:11 (35927): No heartbeat from core client for 30 sec - exiting 15:57:12 (35927): No heartbeat from core client for 30 sec - exiting 15:57:13 (35927): No heartbeat from core client for 30 sec - exiting 15:57:14 (35927): No heartbeat from core client for 30 sec - exiting 15:57:15 (35927): No heartbeat from core client for 30 sec - exiting 15:57:16 (35927): No heartbeat from core client for 30 sec - exiting 15:57:17 (35927): No heartbeat from core client for 30 sec - exiting 15:57:18 (35927): No heartbeat from core client for 30 sec - exiting 15:57:19 (35927): No heartbeat from core client for 30 sec - exiting 15:57:20 (35927): No heartbeat from core client for 30 sec - exiting 15:57:21 (35927): No heartbeat from core client for 30 sec - exiting 15:57:22 (35927): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 16:01:27 (36075): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:58 (36215): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:48 (36710): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:49 (36710): No heartbeat from core client for 30 sec - exiting 16:50:50 (36710): No heartbeat from core client for 30 sec - exiting 16:50:51 (36710): No heartbeat from core client for 30 sec - exiting 16:50:52 (36710): No heartbeat from core client for 30 sec - exiting 16:50:53 (36710): No heartbeat from core client for 30 sec - exiting 16:50:54 (36710): No heartbeat from core client for 30 sec - exiting 16:50:55 (36710): No heartbeat from core client for 30 sec - exiting 16:50:56 (36710): No heartbeat from core client for 30 sec - exiting 16:50:57 (36710): No heartbeat from core client for 30 sec - exiting 16:50:58 (36710): No heartbeat from core client for 30 sec - exiting 16:55:00 (36857): No heartbeat from core client for 30 sec - exiting 16:55:18 (36857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:26 (37014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:51 (39114): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:11 (39337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:12 (39337): No heartbeat from core client for 30 sec - exiting 21:10:13 (39337): No heartbeat from core client for 30 sec - exiting 21:10:14 (39337): No heartbeat from core client for 30 sec - exiting 21:10:15 (39337): No heartbeat from core client for 30 sec - exiting 21:10:16 (39337): No heartbeat from core client for 30 sec - exiting 21:10:17 (39337): No heartbeat from core client for 30 sec - exiting 21:10:18 (39337): No heartbeat from core client for 30 sec - exiting 21:10:19 (39337): No heartbeat from core client for 30 sec - exiting 21:10:20 (39337): No heartbeat from core client for 30 sec - exiting 21:10:21 (39337): No heartbeat from core client for 30 sec - exiting 21:10:22 (39337): No heartbeat from core client for 30 sec - exiting 21:10:23 (39337): No heartbeat from core client for 30 sec - exiting 21:10:24 (39337): No heartbeat from core client for 30 sec - exiting 21:10:25 (39337): No heartbeat from core client for 30 sec - exiting 21:10:26 (39337): No heartbeat from core client for 30 sec - exiting 21:10:27 (39337): No heartbeat from core client for 30 sec - exiting 21:10:28 (39337): No heartbeat from core client for 30 sec - exiting 21:14:31 (39488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:14:32 (39488): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:28:20 (39640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:21 (39640): No heartbeat from core client for 30 sec - exiting 22:28:22 (39640): No heartbeat from core client for 30 sec - exiting 22:28:23 (39640): No heartbeat from core client for 30 sec - exiting 22:28:24 (39640): No heartbeat from core client for 30 sec - exiting 22:28:25 (39640): No heartbeat from core client for 30 sec - exiting 22:28:26 (39640): No heartbeat from core client for 30 sec - exiting 22:28:27 (39640): No heartbeat from core client for 30 sec - exiting 22:28:28 (39640): No heartbeat from core client for 30 sec - exiting 22:28:29 (39640): No heartbeat from core client for 30 sec - exiting 22:32:17 (40378): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:18 (40378): No heartbeat from core client for 30 sec - exiting 22:32:19 (40378): No heartbeat from core client for 30 sec - exiting 22:32:20 (40378): No heartbeat from core client for 30 sec - exiting 22:32:21 (40378): No heartbeat from core client for 30 sec - exiting 22:32:22 (40378): No heartbeat from core client for 30 sec - exiting 22:32:23 (40378): No heartbeat from core client for 30 sec - exiting 22:32:24 (40378): No heartbeat from core client for 30 sec - exiting 22:32:25 (40378): No heartbeat from core client for 30 sec - exiting 22:32:26 (40378): No heartbeat from core client for 30 sec - exiting 22:32:27 (40378): No heartbeat from core client for 30 sec - exiting 22:32:28 (40378): No heartbeat from core client for 30 sec - exiting 22:32:29 (40378): No heartbeat from core client for 30 sec - exiting 22:32:30 (40378): No heartbeat from core client for 30 sec - exiting 22:32:31 (40378): No heartbeat from core client for 30 sec - exiting 22:32:32 (40378): No heartbeat from core client for 30 sec - exiting 22:32:33 (40378): No heartbeat from core client for 30 sec - exiting 23:07:15 (40538): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:01 (40931): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:02 (40931): No heartbeat from core client for 30 sec - exiting 23:11:03 (40931): No heartbeat from core client for 30 sec - exiting 23:11:04 (40931): No heartbeat from core client for 30 sec - exiting 23:11:05 (40931): No heartbeat from core client for 30 sec - exiting 23:11:06 (40931): No heartbeat from core client for 30 sec - exiting 23:11:07 (40931): No heartbeat from core client for 30 sec - exiting 23:11:08 (40931): No heartbeat from core client for 30 sec - exiting 23:11:09 (40931): No heartbeat from core client for 30 sec - exiting 23:11:10 (40931): No heartbeat from core client for 30 sec - exiting 23:11:11 (40931): No heartbeat from core client for 30 sec - exiting 23:11:12 (40931): No heartbeat from core client for 30 sec - exiting 23:11:13 (40931): No heartbeat from core client for 30 sec - exiting 23:11:14 (40931): No heartbeat from core client for 30 sec - exiting 23:11:15 (40931): No heartbeat from core client for 30 sec - exiting 23:11:16 (40931): No heartbeat from core client for 30 sec - exiting 23:11:17 (40931): No heartbeat from core client for 30 sec - exiting 23:11:18 (40931): No heartbeat from core client for 30 sec - exiting 23:11:19 (40931): No heartbeat from core client for 30 sec - exiting 23:11:20 (40931): No heartbeat from core client for 30 sec - exiting 23:11:21 (40931): No heartbeat from core client for 30 sec - exiting 23:11:22 (40931): No heartbeat from core client for 30 sec - exiting 23:11:23 (40931): No heartbeat from core client for 30 sec - exiting 23:11:24 (40931): No heartbeat from core client for 30 sec - exiting 23:11:25 (40931): No heartbeat from core client for 30 sec - exiting 23:11:26 (40931): No heartbeat from core client for 30 sec - exiting 23:11:27 (40931): No heartbeat from core client for 30 sec - exiting 23:11:28 (40931): No heartbeat from core client for 30 sec - exiting 23:11:29 (40931): No heartbeat from core client for 30 sec - exiting 00:50:21 (41079): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:03 (41954): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:04 (41954): No heartbeat from core client for 30 sec - exiting 01:14:20 (42233): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:21 (42233): No heartbeat from core client for 30 sec - exiting 01:14:22 (42233): No heartbeat from core client for 30 sec - exiting 01:14:23 (42233): No heartbeat from core client for 30 sec - exiting 01:14:24 (42233): No heartbeat from core client for 30 sec - exiting 01:14:25 (42233): No heartbeat from core client for 30 sec - exiting 01:14:26 (42233): No heartbeat from core client for 30 sec - exiting 01:14:27 (42233): No heartbeat from core client for 30 sec - exiting 01:14:28 (42233): No heartbeat from core client for 30 sec - exiting 01:14:29 (42233): No heartbeat from core client for 30 sec - exiting 01:14:30 (42233): No heartbeat from core client for 30 sec - exiting 01:14:31 (42233): No heartbeat from core client for 30 sec - exiting 01:14:32 (42233): No heartbeat from core client for 30 sec - exiting 01:14:33 (42233): No heartbeat from core client for 30 sec - exiting 01:14:34 (42233): No heartbeat from core client for 30 sec - exiting 01:18:19 (42372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:20 (42372): No heartbeat from core client for 30 sec - exiting 01:18:21 (42372): No heartbeat from core client for 30 sec - exiting 01:18:22 (42372): No heartbeat from core client for 30 sec - exiting 01:18:23 (42372): No heartbeat from core client for 30 sec - exiting 01:18:24 (42372): No heartbeat from core client for 30 sec - exiting 01:18:25 (42372): No heartbeat from core client for 30 sec - exiting 01:18:26 (42372): No heartbeat from core client for 30 sec - exiting 01:18:27 (42372): No heartbeat from core client for 30 sec - exiting 01:18:28 (42372): No heartbeat from core client for 30 sec - exiting 01:18:29 (42372): No heartbeat from core client for 30 sec - exiting 01:18:30 (42372): No heartbeat from core client for 30 sec - exiting 01:18:31 (42372): No heartbeat from core client for 30 sec - exiting 01:22:26 (42552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:27 (42552): No heartbeat from core client for 30 sec - exiting 01:22:28 (42552): No heartbeat from core client for 30 sec - exiting 01:22:29 (42552): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 05:34:53 (42696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:34:54 (42696): No heartbeat from core client for 30 sec - exiting 05:34:55 (42696): No heartbeat from core client for 30 sec - exiting 05:34:56 (42696): No heartbeat from core client for 30 sec - exiting 05:34:57 (42696): No heartbeat from core client for 30 sec - exiting 05:39:16 (44919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:44:10 (45069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:49:14 (45246): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:27 (49125): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:09 (49271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:24 (49421): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:18 (53880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:19 (53880): No heartbeat from core client for 30 sec - exiting 21:42:20 (53880): No heartbeat from core client for 30 sec - exiting 21:42:21 (53880): No heartbeat from core client for 30 sec - exiting 21:42:22 (53880): No heartbeat from core client for 30 sec - exiting 21:42:23 (53880): No heartbeat from core client for 30 sec - exiting 21:42:24 (53880): No heartbeat from core client for 30 sec - exiting 21:42:25 (53880): No heartbeat from core client for 30 sec - exiting 21:42:26 (53880): No heartbeat from core client for 30 sec - exiting 21:42:27 (53880): No heartbeat from core client for 30 sec - exiting 21:42:28 (53880): No heartbeat from core client for 30 sec - exiting 21:42:29 (53880): No heartbeat from core client for 30 sec - exiting 21:58:25 (54056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:02:06 (54305): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:02:07 (54305): No heartbeat from core client for 30 sec - exiting 22:02:08 (54305): No heartbeat from core client for 30 sec - exiting 22:02:09 (54305): No heartbeat from core client for 30 sec - exiting 22:02:10 (54305): No heartbeat from core client for 30 sec - exiting 22:02:11 (54305): No heartbeat from core client for 30 sec - exiting 22:02:12 (54305): No heartbeat from core client for 30 sec - exiting 22:02:13 (54305): No heartbeat from core client for 30 sec - exiting 22:02:14 (54305): No heartbeat from core client for 30 sec - exiting 22:02:15 (54305): No heartbeat from core client for 30 sec - exiting 22:02:16 (54305): No heartbeat from core client for 30 sec - exiting 22:02:17 (54305): No heartbeat from core client for 30 sec - exiting 22:02:18 (54305): No heartbeat from core client for 30 sec - exiting 22:02:19 (54305): No heartbeat from core client for 30 sec - exiting 22:02:20 (54305): No heartbeat from core client for 30 sec - exiting 22:02:21 (54305): No heartbeat from core client for 30 sec - exiting 22:02:22 (54305): No heartbeat from core client for 30 sec - exiting 22:02:23 (54305): No heartbeat from core client for 30 sec - exiting 22:02:24 (54305): No heartbeat from core client for 30 sec - exiting 22:02:25 (54305): No heartbeat from core client for 30 sec - exiting 22:06:18 (54463): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:19 (54463): No heartbeat from core client for 30 sec - exiting 22:06:20 (54463): No heartbeat from core client for 30 sec - exiting 22:06:21 (54463): No heartbeat from core client for 30 sec - exiting 22:06:22 (54463): No heartbeat from core client for 30 sec - exiting 22:06:23 (54463): No heartbeat from core client for 30 sec - exiting 22:06:24 (54463): No heartbeat from core client for 30 sec - exiting 22:06:25 (54463): No heartbeat from core client for 30 sec - exiting 22:06:26 (54463): No heartbeat from core client for 30 sec - exiting 22:06:27 (54463): No heartbeat from core client for 30 sec - exiting 22:06:28 (54463): No heartbeat from core client for 30 sec - exiting 22:06:29 (54463): No heartbeat from core client for 30 sec - exiting 22:06:30 (54463): No heartbeat from core client for 30 sec - exiting 22:06:31 (54463): No heartbeat from core client for 30 sec - exiting 22:06:32 (54463): No heartbeat from core client for 30 sec - exiting 22:06:33 (54463): No heartbeat from core client for 30 sec - exiting 22:06:34 (54463): No heartbeat from core client for 30 sec - exiting 22:10:59 (54625): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:00 (54625): No heartbeat from core client for 30 sec - exiting 22:11:01 (54625): No heartbeat from core client for 30 sec - exiting 22:11:02 (54625): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jun 2013 21:47:47	1282401	15836087	hadcm3n_o723_1980_40_008385985_1	51,840	128,665	2.4820
10 Jun 2013 02:25:08	1282401	15836087	hadcm3n_o723_1980_40_008385985_1	25,920	65,601	2.5309