Task 15835743

Name	hadcm3n_u5mj_2020_40_008338527_3
Workunit	8489388
Created	8 Jun 2013, 21:12:27 UTC
Sent	8 Jun 2013, 21:28:02 UTC
Report deadline	8 Sep 2013, 4:55:13 UTC
Received	10 Jun 2013, 5:58:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1282401
Run time	1 days 3 hours 50 min 53 sec
CPU time	1 days 3 hours 5 min 55 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 23:31:05 (25106): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:04 (25499): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:24:51 (25893): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:28:43 (26642): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:28:44 (26642): No heartbeat from core client for 30 sec - exiting 01:32:21 (26797): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:32:59 (26797): No heartbeat from core client for 30 sec - exiting 01:37:43 (26964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:29 (27142): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:44 (27351): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:15 (27523): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:46 (27670): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:47 (27670): No heartbeat from core client for 30 sec - exiting 01:57:48 (27670): No heartbeat from core client for 30 sec - exiting 01:57:49 (27670): No heartbeat from core client for 30 sec - exiting 01:57:50 (27670): No heartbeat from core client for 30 sec - exiting 13:50:51 (27831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:20 (34180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:40 (34339): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:04 (34507): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:17 (35422): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:18 (35422): No heartbeat from core client for 30 sec - exiting 15:40:19 (35422): No heartbeat from core client for 30 sec - exiting 15:40:20 (35422): No heartbeat from core client for 30 sec - exiting 15:40:21 (35422): No heartbeat from core client for 30 sec - exiting 15:40:22 (35422): No heartbeat from core client for 30 sec - exiting 15:40:23 (35422): No heartbeat from core client for 30 sec - exiting 15:40:24 (35422): No heartbeat from core client for 30 sec - exiting 15:40:25 (35422): No heartbeat from core client for 30 sec - exiting 15:40:26 (35422): No heartbeat from core client for 30 sec - exiting 15:40:27 (35422): No heartbeat from core client for 30 sec - exiting 15:40:28 (35422): No heartbeat from core client for 30 sec - exiting 15:53:16 (35580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:57:07 (35854): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:26 (36001): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:58 (36138): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:48 (36657): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:49 (36657): No heartbeat from core client for 30 sec - exiting 16:50:50 (36657): No heartbeat from core client for 30 sec - exiting 16:50:51 (36657): No heartbeat from core client for 30 sec - exiting 16:50:52 (36657): No heartbeat from core client for 30 sec - exiting 16:50:53 (36657): No heartbeat from core client for 30 sec - exiting 16:50:54 (36657): No heartbeat from core client for 30 sec - exiting 16:50:55 (36657): No heartbeat from core client for 30 sec - exiting 16:50:56 (36657): No heartbeat from core client for 30 sec - exiting 16:50:57 (36657): No heartbeat from core client for 30 sec - exiting 16:55:00 (36802): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:26 (36955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:51 (39066): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:52 (39066): No heartbeat from core client for 30 sec - exiting 21:05:53 (39066): No heartbeat from core client for 30 sec - exiting 21:05:54 (39066): No heartbeat from core client for 30 sec - exiting 21:05:55 (39066): No heartbeat from core client for 30 sec - exiting 21:05:56 (39066): No heartbeat from core client for 30 sec - exiting 21:05:57 (39066): No heartbeat from core client for 30 sec - exiting 21:05:58 (39066): No heartbeat from core client for 30 sec - exiting 21:05:59 (39066): No heartbeat from core client for 30 sec - exiting 21:06:00 (39066): No heartbeat from core client for 30 sec - exiting 21:06:01 (39066): No heartbeat from core client for 30 sec - exiting 21:06:02 (39066): No heartbeat from core client for 30 sec - exiting 21:10:10 (39288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:14:31 (39438): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:19 (39590): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:20 (39590): No heartbeat from core client for 30 sec - exiting 22:28:21 (39590): No heartbeat from core client for 30 sec - exiting 22:28:22 (39590): No heartbeat from core client for 30 sec - exiting 22:28:23 (39590): No heartbeat from core client for 30 sec - exiting 22:28:24 (39590): No heartbeat from core client for 30 sec - exiting 22:28:25 (39590): No heartbeat from core client for 30 sec - exiting 22:28:26 (39590): No heartbeat from core client for 30 sec - exiting 22:28:27 (39590): No heartbeat from core client for 30 sec - exiting 22:28:28 (39590): No heartbeat from core client for 30 sec - exiting 22:28:29 (39590): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:32:17 (40326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:18 (40326): No heartbeat from core client for 30 sec - exiting 22:32:19 (40326): No heartbeat from core client for 30 sec - exiting 23:07:16 (40479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:00 (40898): No heartbeat from core client for 30 sec - exiting 23:11:01 (40898): No heartbeat from core client for 30 sec - exiting 23:11:02 (40898): No heartbeat from core client for 30 sec - exiting 23:11:03 (40898): No heartbeat from core client for 30 sec - exiting 23:11:04 (40898): No heartbeat from core client for 30 sec - exiting 23:11:05 (40898): No heartbeat from core client for 30 sec - exiting 23:11:06 (40898): No heartbeat from core client for 30 sec - exiting 23:11:07 (40898): No heartbeat from core client for 30 sec - exiting 23:11:08 (40898): No heartbeat from core client for 30 sec - exiting 23:11:09 (40898): No heartbeat from core client for 30 sec - exiting 23:11:10 (40898): No heartbeat from core client for 30 sec - exiting 23:11:11 (40898): No heartbeat from core client for 30 sec - exiting 23:11:12 (40898): No heartbeat from core client for 30 sec - exiting 23:11:13 (40898): No heartbeat from core client for 30 sec - exiting 23:11:14 (40898): No heartbeat from core client for 30 sec - exiting 23:11:15 (40898): No heartbeat from core client for 30 sec - exiting 23:11:16 (40898): No heartbeat from core client for 30 sec - exiting 23:11:17 (40898): No heartbeat from core client for 30 sec - exiting 23:11:18 (40898): No heartbeat from core client for 30 sec - exiting 23:11:19 (40898): No heartbeat from core client for 30 sec - exiting 23:11:20 (40898): No heartbeat from core client for 30 sec - exiting 23:11:21 (40898): No heartbeat from core client for 30 sec - exiting 23:11:22 (40898): No heartbeat from core client for 30 sec - exiting 23:11:23 (40898): No heartbeat from core client for 30 sec - exiting 23:11:24 (40898): No heartbeat from core client for 30 sec - exiting 23:11:25 (40898): No heartbeat from core client for 30 sec - exiting 23:11:26 (40898): No heartbeat from core client for 30 sec - exiting 23:11:27 (40898): No heartbeat from core client for 30 sec - exiting 23:11:28 (40898): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:21 (41045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:03 (41944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:20 (42225): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:18 (42363): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:18:19 (42363): No heartbeat from core client for 30 sec - exiting 01:18:20 (42363): No heartbeat from core client for 30 sec - exiting 01:18:21 (42363): No heartbeat from core client for 30 sec - exiting 01:18:22 (42363): No heartbeat from core client for 30 sec - exiting 01:18:23 (42363): No heartbeat from core client for 30 sec - exiting 01:18:24 (42363): No heartbeat from core client for 30 sec - exiting 01:18:25 (42363): No heartbeat from core client for 30 sec - exiting 01:18:26 (42363): No heartbeat from core client for 30 sec - exiting 01:18:27 (42363): No heartbeat from core client for 30 sec - exiting 01:18:28 (42363): No heartbeat from core client for 30 sec - exiting 01:18:29 (42363): No heartbeat from core client for 30 sec - exiting 01:18:30 (42363): No heartbeat from core client for 30 sec - exiting 01:22:26 (42544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:27 (42544): No heartbeat from core client for 30 sec - exiting 01:22:28 (42544): No heartbeat from core client for 30 sec - exiting 01:22:29 (42544): No heartbeat from core client for 30 sec - exiting 05:34:53 (42687): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:34:54 (42687): No heartbeat from core client for 30 sec - exiting 05:34:55 (42687): No heartbeat from core client for 30 sec - exiting 05:34:56 (42687): No heartbeat from core client for 30 sec - exiting 05:34:57 (42687): No heartbeat from core client for 30 sec - exiting 05:34:58 (42687): No heartbeat from core client for 30 sec - exiting 05:39:17 (44910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:18 (44910): No heartbeat from core client for 30 sec - exiting 05:39:19 (44910): No heartbeat from core client for 30 sec - exiting 05:39:20 (44910): No heartbeat from core client for 30 sec - exiting 05:39:21 (44910): No heartbeat from core client for 30 sec - exiting 05:39:22 (44910): No heartbeat from core client for 30 sec - exiting 05:39:23 (44910): No heartbeat from core client for 30 sec - exiting 05:39:24 (44910): No heartbeat from core client for 30 sec - exiting 05:39:25 (44910): No heartbeat from core client for 30 sec - exiting 05:39:26 (44910): No heartbeat from core client for 30 sec - exiting 05:39:27 (44910): No heartbeat from core client for 30 sec - exiting 05:39:28 (44910): No heartbeat from core client for 30 sec - exiting 05:39:29 (44910): No heartbeat from core client for 30 sec - exiting 05:39:30 (44910): No heartbeat from core client for 30 sec - exiting 05:39:31 (44910): No heartbeat from core client for 30 sec - exiting 05:39:32 (44910): No heartbeat from core client for 30 sec - exiting 05:39:33 (44910): No heartbeat from core client for 30 sec - exiting 05:39:34 (44910): No heartbeat from core client for 30 sec - exiting 05:39:35 (44910): No heartbeat from core client for 30 sec - exiting 05:39:36 (44910): No heartbeat from core client for 30 sec - exiting 05:39:37 (44910): No heartbeat from core client for 30 sec - exiting 05:39:38 (44910): No heartbeat from core client for 30 sec - exiting 05:39:39 (44910): No heartbeat from core client for 30 sec - exiting 05:39:40 (44910): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a8400] [0xf77a8425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75c51df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75c8825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75b04d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=45061, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a4400] [0xf77a4425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75c11df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75c4825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75ac4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=45061, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77ae400] [0xf77ae425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75cb1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75ce825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75b64d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=45061, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf770e400] [0xf770e425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf752b1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf752e825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75164d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=45061, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a4400] [0xf77a4425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75c11df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75c4825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75ac4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=45061, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76f7400] [0xf76f7425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75141df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7517825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74ff4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=45061, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Jun 2013 17:13:27	1282401	15835743	hadcm3n_u5mj_2020_40_008338527_3	25,920	59,688	2.3028