Task 15770500

Name	hadcm3n_487r_1940_40_008310518_1
Workunit	8461653
Created	10 May 2013, 8:47:46 UTC
Sent	10 May 2013, 8:48:13 UTC
Report deadline	9 Aug 2013, 16:15:24 UTC
Received	11 May 2013, 6:21:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1282911
Run time	18 hours 49 min 43 sec
CPU time	16 hours 39 min 57 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 00:55:16 (10148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:36 (11702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:44 (11970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:31 (12093): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:32 (12093): No heartbeat from core client for 30 sec - exiting 01:07:33 (12093): No heartbeat from core client for 30 sec - exiting 01:07:34 (12093): No heartbeat from core client for 30 sec - exiting 01:07:35 (12093): No heartbeat from core client for 30 sec - exiting 01:07:36 (12093): No heartbeat from core client for 30 sec - exiting 01:07:37 (12093): No heartbeat from core client for 30 sec - exiting 01:07:38 (12093): No heartbeat from core client for 30 sec - exiting 01:07:39 (12093): No heartbeat from core client for 30 sec - exiting 01:07:40 (12093): No heartbeat from core client for 30 sec - exiting 01:07:41 (12093): No heartbeat from core client for 30 sec - exiting 01:07:42 (12093): No heartbeat from core client for 30 sec - exiting 01:07:43 (12093): No heartbeat from core client for 30 sec - exiting 01:07:44 (12093): No heartbeat from core client for 30 sec - exiting 01:07:45 (12093): No heartbeat from core client for 30 sec - exiting 01:07:46 (12093): No heartbeat from core client for 30 sec - exiting 01:07:47 (12093): No heartbeat from core client for 30 sec - exiting 01:07:48 (12093): No heartbeat from core client for 30 sec - exiting 01:07:49 (12093): No heartbeat from core client for 30 sec - exiting 01:12:07 (12186): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:15 (12292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:16 (12292): No heartbeat from core client for 30 sec - exiting 01:16:17 (12292): No heartbeat from core client for 30 sec - exiting 01:16:18 (12292): No heartbeat from core client for 30 sec - exiting 01:16:19 (12292): No heartbeat from core client for 30 sec - exiting 01:16:20 (12292): No heartbeat from core client for 30 sec - exiting 01:16:21 (12292): No heartbeat from core client for 30 sec - exiting 01:20:15 (12384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:16 (12384): No heartbeat from core client for 30 sec - exiting 01:20:17 (12384): No heartbeat from core client for 30 sec - exiting 01:20:18 (12384): No heartbeat from core client for 30 sec - exiting 01:20:19 (12384): No heartbeat from core client for 30 sec - exiting 01:20:20 (12384): No heartbeat from core client for 30 sec - exiting 01:20:21 (12384): No heartbeat from core client for 30 sec - exiting 01:20:22 (12384): No heartbeat from core client for 30 sec - exiting 01:20:23 (12384): No heartbeat from core client for 30 sec - exiting 03:22:16 (12485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:25 (12619): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:10 (12725): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:11 (12725): No heartbeat from core client for 30 sec - exiting 03:30:12 (12725): No heartbeat from core client for 30 sec - exiting 03:30:13 (12725): No heartbeat from core client for 30 sec - exiting 03:30:14 (12725): No heartbeat from core client for 30 sec - exiting 03:30:15 (12725): No heartbeat from core client for 30 sec - exiting 03:30:16 (12725): No heartbeat from core client for 30 sec - exiting 03:30:17 (12725): No heartbeat from core client for 30 sec - exiting 03:34:08 (12809): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:09 (12809): No heartbeat from core client for 30 sec - exiting 03:34:10 (12809): No heartbeat from core client for 30 sec - exiting 03:34:11 (12809): No heartbeat from core client for 30 sec - exiting 03:34:12 (12809): No heartbeat from core client for 30 sec - exiting 03:34:13 (12809): No heartbeat from core client for 30 sec - exiting 03:34:14 (12809): No heartbeat from core client for 30 sec - exiting 03:34:15 (12809): No heartbeat from core client for 30 sec - exiting 03:34:16 (12809): No heartbeat from core client for 30 sec - exiting 03:34:17 (12809): No heartbeat from core client for 30 sec - exiting 03:34:18 (12809): No heartbeat from core client for 30 sec - exiting 03:41:30 (12893): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:36 (12974): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:39 (13057): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:26 (13141): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:45 (13221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:33 (13352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:38 (13352): No heartbeat from core client for 30 sec - exiting 04:24:39 (13352): No heartbeat from core client for 30 sec - exiting 04:28:35 (13432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 04:36:44 (13524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Atmos Hold Restart file rename failed on atmos_restart.hold 04:44:29 (13643): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 May 2013 04:05:57	1281428	15770500	hadcm3n_487r_1940_40_008310518_1	25,920	59,436	2.2931