Task 15775295

Name	hadcm3n_4g6m_1980_40_008365428_0
Workunit	8516287
Created	11 May 2013, 0:56:28 UTC
Sent	11 May 2013, 1:02:50 UTC
Report deadline	10 Aug 2013, 8:30:01 UTC
Received	4 Sep 2013, 15:33:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291528
Run time	2 days 17 hours 15 min 43 sec
CPU time	2 days 13 hours 17 min 57 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 03:22:16 (12573): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:25 (12671): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:10 (12769): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:31 (12865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:27 (12954): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:26 (13037): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:27 (13037): No heartbeat from core client for 30 sec - exiting 03:49:28 (13037): No heartbeat from core client for 30 sec - exiting 03:49:29 (13037): No heartbeat from core client for 30 sec - exiting 03:49:30 (13037): No heartbeat from core client for 30 sec - exiting 03:49:31 (13037): No heartbeat from core client for 30 sec - exiting 03:49:32 (13037): No heartbeat from core client for 30 sec - exiting 03:49:33 (13037): No heartbeat from core client for 30 sec - exiting 03:49:34 (13037): No heartbeat from core client for 30 sec - exiting 03:49:35 (13037): No heartbeat from core client for 30 sec - exiting 03:49:36 (13037): No heartbeat from core client for 30 sec - exiting 03:49:37 (13037): No heartbeat from core client for 30 sec - exiting 03:49:38 (13037): No heartbeat from core client for 30 sec - exiting 03:53:27 (13121): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:28 (13121): No heartbeat from core client for 30 sec - exiting 03:53:29 (13121): No heartbeat from core client for 30 sec - exiting 03:53:30 (13121): No heartbeat from core client for 30 sec - exiting 03:53:31 (13121): No heartbeat from core client for 30 sec - exiting 04:20:44 (13205): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:45 (13205): No heartbeat from core client for 30 sec - exiting 04:24:33 (13336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:35 (13424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:33:08 (13516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:44 (13592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:30 (13691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:10 (13775): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:11 (13775): No heartbeat from core client for 30 sec - exiting 06:47:12 (13775): No heartbeat from core client for 30 sec - exiting 06:47:13 (13775): No heartbeat from core client for 30 sec - exiting 06:47:14 (13775): No heartbeat from core client for 30 sec - exiting 06:47:15 (13775): No heartbeat from core client for 30 sec - exiting 06:47:16 (13775): No heartbeat from core client for 30 sec - exiting 06:51:09 (13934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:18 (14008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:19 (14008): No heartbeat from core client for 30 sec - exiting 00:12:20 (14008): No heartbeat from core client for 30 sec - exiting 00:12:21 (14008): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 05:55:34 (14874): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:35 (15067): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:36 (15067): No heartbeat from core client for 30 sec - exiting 05:59:37 (15067): No heartbeat from core client for 30 sec - exiting 05:59:38 (15067): No heartbeat from core client for 30 sec - exiting 06:03:29 (15190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:30 (15190): No heartbeat from core client for 30 sec - exiting 06:03:31 (15190): No heartbeat from core client for 30 sec - exiting 18:06:18 (15297): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:58 (16004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:43 (16080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:44 (16080): No heartbeat from core client for 30 sec - exiting 18:13:45 (16080): No heartbeat from core client for 30 sec - exiting 18:13:46 (16080): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 18:17:25 (16160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:14 (16243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:15 (16243): No heartbeat from core client for 30 sec - exiting 18:21:16 (16243): No heartbeat from core client for 30 sec - exiting 18:21:17 (16243): No heartbeat from core client for 30 sec - exiting 18:21:18 (16243): No heartbeat from core client for 30 sec - exiting 18:21:19 (16243): No heartbeat from core client for 30 sec - exiting 18:21:20 (16243): No heartbeat from core client for 30 sec - exiting 18:21:21 (16243): No heartbeat from core client for 30 sec - exiting 18:21:22 (16243): No heartbeat from core client for 30 sec - exiting 18:21:23 (16243): No heartbeat from core client for 30 sec - exiting 18:21:24 (16243): No heartbeat from core client for 30 sec - exiting 18:21:25 (16243): No heartbeat from core client for 30 sec - exiting 18:21:26 (16243): No heartbeat from core client for 30 sec - exiting 18:25:12 (16323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:29:06 (16403): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:29:12 (16403): No heartbeat from core client for 30 sec - exiting 18:32:55 (16485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:45 (16573): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:46 (16573): No heartbeat from core client for 30 sec - exiting 18:36:47 (16573): No heartbeat from core client for 30 sec - exiting 18:36:48 (16573): No heartbeat from core client for 30 sec - exiting 18:36:49 (16573): No heartbeat from core client for 30 sec - exiting 18:36:50 (16573): No heartbeat from core client for 30 sec - exiting 18:36:51 (16573): No heartbeat from core client for 30 sec - exiting 18:40:33 (16653): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:55 (16713): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:21 (16874): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:22 (16874): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:48:57 (3914): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:10 (4302): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:11 (4302): No heartbeat from core client for 30 sec - exiting 16:23:12 (4302): No heartbeat from core client for 30 sec - exiting 16:23:13 (4302): No heartbeat from core client for 30 sec - exiting 16:23:14 (4302): No heartbeat from core client for 30 sec - exiting 16:39:51 (4829): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:09 (4935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:10 (4935): No heartbeat from core client for 30 sec - exiting 16:42:11 (4935): No heartbeat from core client for 30 sec - exiting 16:42:12 (4935): No heartbeat from core client for 30 sec - exiting 16:42:13 (4935): No heartbeat from core client for 30 sec - exiting 16:42:14 (4935): No heartbeat from core client for 30 sec - exiting 16:42:15 (4935): No heartbeat from core client for 30 sec - exiting 16:42:16 (4935): No heartbeat from core client for 30 sec - exiting 16:42:17 (4935): No heartbeat from core client for 30 sec - exiting 16:42:18 (4935): No heartbeat from core client for 30 sec - exiting 16:42:19 (4935): No heartbeat from core client for 30 sec - exiting 16:42:20 (4935): No heartbeat from core client for 30 sec - exiting 16:42:21 (4935): No heartbeat from core client for 30 sec - exiting 16:42:22 (4935): No heartbeat from core client for 30 sec - exiting 16:42:23 (4935): No heartbeat from core client for 30 sec - exiting 16:42:24 (4935): No heartbeat from core client for 30 sec - exiting 16:42:25 (4935): No heartbeat from core client for 30 sec - exiting 16:48:20 (5045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:59 (5150): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:08 (5264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:46 (5349): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:37 (5650): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:38 (5650): No heartbeat from core client for 30 sec - exiting 17:49:39 (5650): No heartbeat from core client for 30 sec - exiting 17:49:40 (5650): No heartbeat from core client for 30 sec - exiting 17:49:41 (5650): No heartbeat from core client for 30 sec - exiting 17:49:42 (5650): No heartbeat from core client for 30 sec - exiting 17:49:43 (5650): No heartbeat from core client for 30 sec - exiting 17:49:44 (5650): No heartbeat from core client for 30 sec - exiting 17:49:45 (5650): No heartbeat from core client for 30 sec - exiting 17:49:46 (5650): No heartbeat from core client for 30 sec - exiting 17:49:47 (5650): No heartbeat from core client for 30 sec - exiting 17:49:48 (5650): No heartbeat from core client for 30 sec - exiting 17:49:49 (5650): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 17:54:20 (35876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 18:05:44 (43168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:11 (43257): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:15:55 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:15:56 (3496): No heartbeat from core client for 30 sec - exiting 13:15:57 (3496): No heartbeat from core client for 30 sec - exiting 13:15:58 (3496): No heartbeat from core client for 30 sec - exiting 13:15:59 (3496): No heartbeat from core client for 30 sec - exiting 13:18:22 (4232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:29:55 (3746): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:57 (3746): No heartbeat from core client for 30 sec - exiting 13:29:58 (3746): No heartbeat from core client for 30 sec - exiting 13:40:00 (4483): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:28:23 (2930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:28:24 (2930): No heartbeat from core client for 30 sec - exiting 14:28:25 (2930): No heartbeat from core client for 30 sec - exiting 14:28:26 (2930): No heartbeat from core client for 30 sec - exiting 15:54:54 (3919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:56 (4413): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:57 (4413): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4506, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4506, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4506, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4506, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4506, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4506, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 May 2013 20:11:56	1282911	15775295	hadcm3n_4g6m_1980_40_008365428_0	77,760	196,488	2.5269
12 May 2013 20:42:15	1281428	15775295	hadcm3n_4g6m_1980_40_008365428_0	51,840	136,788	2.6387
12 May 2013 01:18:17	1281428	15775295	hadcm3n_4g6m_1980_40_008365428_0	25,920	76,161	2.9383