Task 15828401

Name	hadcm3n_zjn9_1960_40_008389680_3
Workunit	8540539
Created	4 Jun 2013, 9:14:34 UTC
Sent	4 Jun 2013, 9:22:58 UTC
Report deadline	3 Sep 2013, 16:50:09 UTC
Received	2 Dec 2013, 23:15:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1269200
Run time	12 days 4 hours 40 min 51 sec
CPU time	12 days 0 hours 23 min 51 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	2.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:24:37 (21900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:20 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:42 (5200): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 16:29:43 (5281): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:25 (5319): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:32:02 (5319): No heartbeat from core client for 30 sec - exiting 16:32:03 (5319): No heartbeat from core client for 30 sec - exiting 16:32:04 (5319): No heartbeat from core client for 30 sec - exiting 16:32:05 (5319): No heartbeat from core client for 30 sec - exiting 16:32:06 (5319): No heartbeat from core client for 30 sec - exiting 16:32:07 (5319): No heartbeat from core client for 30 sec - exiting 16:32:08 (5319): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 16:34:10 (5374): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:35:34 (5394): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:35:40 (5394): No heartbeat from core client for 30 sec - exiting 16:35:41 (5394): No heartbeat from core client for 30 sec - exiting 16:36:32 (5414): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:38:34 (5423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:02 (5423): No heartbeat from core client for 30 sec - exiting 16:39:03 (5423): No heartbeat from core client for 30 sec - exiting 16:39:04 (5423): No heartbeat from core client for 30 sec - exiting 16:39:05 (5423): No heartbeat from core client for 30 sec - exiting 16:42:21 (5450): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:22 (5450): No heartbeat from core client for 30 sec - exiting 16:42:23 (5450): No heartbeat from core client for 30 sec - exiting 16:42:24 (5450): No heartbeat from core client for 30 sec - exiting 16:45:11 (5491): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:52:17 (5547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:55:33 (5622): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:43 (5622): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 16:59:43 (5657): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:47 (5759): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:23 (5770): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:16:41 (5883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:00:16 (5962): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:29 (6295): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:08:25 (6340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:08:26 (6340): No heartbeat from core client for 30 sec - exiting 18:08:27 (6340): No heartbeat from core client for 30 sec - exiting 18:15:02 (6367): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:18:53 (6468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:54 (6468): No heartbeat from core client for 30 sec - exiting 18:18:55 (6468): No heartbeat from core client for 30 sec - exiting 18:20:10 (6511): No heartbeat from core client for 30 sec - exiting 18:20:12 (6511): No heartbeat from core client for 30 sec - exiting 18:20:13 (6511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:29:19 (6562): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:29:21 (6562): No heartbeat from core client for 30 sec - exiting 18:31:52 (6648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:43:56 (25559): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:48:26 (18526): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:34 (18592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:37 (18592): No heartbeat from core client for 30 sec - exiting 20:53:38 (18592): No heartbeat from core client for 30 sec - exiting 20:53:39 (18592): No heartbeat from core client for 30 sec - exiting 20:53:40 (18592): No heartbeat from core client for 30 sec - exiting 20:53:41 (18592): No heartbeat from core client for 30 sec - exiting 20:53:42 (18592): No heartbeat from core client for 30 sec - exiting 20:53:43 (18592): No heartbeat from core client for 30 sec - exiting 20:53:44 (18592): No heartbeat from core client for 30 sec - exiting 20:53:45 (18592): No heartbeat from core client for 30 sec - exiting 20:53:46 (18592): No heartbeat from core client for 30 sec - exiting 20:53:47 (18592): No heartbeat from core client for 30 sec - exiting 20:53:48 (18592): No heartbeat from core client for 30 sec - exiting 20:53:49 (18592): No heartbeat from core client for 30 sec - exiting 20:53:50 (18592): No heartbeat from core client for 30 sec - exiting 20:53:51 (18592): No heartbeat from core client for 30 sec - exiting 20:53:52 (18592): No heartbeat from core client for 30 sec - exiting 20:53:53 (18592): No heartbeat from core client for 30 sec - exiting 20:53:54 (18592): No heartbeat from core client for 30 sec - exiting 20:53:55 (18592): No heartbeat from core client for 30 sec - exiting 20:53:56 (18592): No heartbeat from core client for 30 sec - exiting 20:53:57 (18592): No heartbeat from core client for 30 sec - exiting 20:53:58 (18592): No heartbeat from core client for 30 sec - exiting 20:53:59 (18592): No heartbeat from core client for 30 sec - exiting 20:54:00 (18592): No heartbeat from core client for 30 sec - exiting 20:54:01 (18592): No heartbeat from core client for 30 sec - exiting 20:54:02 (18592): No heartbeat from core client for 30 sec - exiting 20:54:03 (18592): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:56:56 (18660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:10 (18660): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:01:41 (18712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:43 (18712): No heartbeat from core client for 30 sec - exiting 21:01:50 (18712): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 21:05:14 (18786): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 21:05:25 (18786): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 01:29:12 (7492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:30:06 (29429): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:35:00 (29511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:40:15 (29691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:16:16 (29782): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:59 (30604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:01 (30604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 02:20:28 (30625): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 02:23:09 (30664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:06 (30706): No heartbeat from core client for 30 sec - exiting 02:24:09 (30706): No heartbeat from core client for 30 sec - exiting 02:24:11 (30706): No heartbeat from core client for 30 sec - exiting 02:24:12 (30706): No heartbeat from core client for 30 sec - exiting 02:24:13 (30706): No heartbeat from core client for 30 sec - exiting 02:24:14 (30706): No heartbeat from core client for 30 sec - exiting 02:24:15 (30706): No heartbeat from core client for 30 sec - exiting 02:24:16 (30706): No heartbeat from core client for 30 sec - exiting 02:24:17 (30706): No heartbeat from core client for 30 sec - exiting 02:24:18 (30706): No heartbeat from core client for 30 sec - exiting 02:24:19 (30706): No heartbeat from core client for 30 sec - exiting 02:24:20 (30706): No heartbeat from core client for 30 sec - exiting 02:24:21 (30706): No heartbeat from core client for 30 sec - exiting 02:24:22 (30706): No heartbeat from core client for 30 sec - exiting 02:24:23 (30706): No heartbeat from core client for 30 sec - exiting 02:24:24 (30706): No heartbeat from core client for 30 sec - exiting 02:24:26 (30706): No heartbeat from core client for 30 sec - exiting 02:24:27 (30706): No heartbeat from core client for 30 sec - exiting 02:24:28 (30706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:29:47 (30765): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:30:32 (30981): No heartbeat from core client for 30 sec - exiting 02:30:33 (30981): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:34:07 (31033): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:09 (31033): No heartbeat from core client for 30 sec - exiting 02:34:10 (31033): No heartbeat from core client for 30 sec - exiting 02:34:11 (31033): No heartbeat from core client for 30 sec - exiting 02:34:12 (31033): No heartbeat from core client for 30 sec - exiting 02:34:13 (31033): No heartbeat from core client for 30 sec - exiting 02:34:14 (31033): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 02:35:55 (31073): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 02:35:57 (31073): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Signal 2 received, exiting... Signal 2 received, exiting... Called boinc_finish 18:20:12 (9937): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:54 (31521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:18 (31535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:06 (31955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:47:34 (32099): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:24 (32298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:52 (32698): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:13 (32726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:17:29 (605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:17:30 (605): No heartbeat from core client for 30 sec - exiting 21:18:07 (866): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:40:04 (895): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:43:56 (1091): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:45:23 (1118): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:46 (1222): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:20:27 (1467): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:03 (1726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:48 (1809): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:41:40 (1919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:47 (1987): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:58 (1987): No heartbeat from core client for 30 sec - exiting 22:43:59 (1987): No heartbeat from core client for 30 sec - exiting 22:49:55 (2017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:56 (2017):23:00:52 (2120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:26 (2271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:13:47 (2378): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:53 (2423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:22 (2489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:34:46 (2519): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:10 (2715): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:13 (2715): No heartbeat from core client for 30 sec - exiting 23:56:36 (2934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:56:38 (2934): No heartbeat from core client for 30 sec - exiting 23:59:40 (2973): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:11:20 (3012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:11:22 (3012): No heartbeat from core client for 30 sec - exiting 00:11:23 (3012): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf774c400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf774c425] /lib/libc.so.6(gsignal+0x4f)[0xf758631f] /lib/libc.so.6(abort+0x143)[0xf7587c03] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75713d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3177, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7770400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf7770425] /lib/libc.so.6(gsignal+0x4f)[0xf75aa31f] /lib/libc.so.6(abort+0x143)[0xf75abc03] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75953d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3177, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cc400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf77cc425] /lib/libc.so.6(gsignal+0x4f)[0xf760631f] /lib/libc.so.6(abort+0x143)[0xf7607c03] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75f13d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3177, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b0400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf77b0425] /lib/libc.so.6(gsignal+0x4f)[0xf75ea31f] /lib/libc.so.6(abort+0x143)[0xf75ebc03] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75d53d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3177, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7786400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf7786425] /lib/libc.so.6(gsignal+0x4f)[0xf75c031f] /lib/libc.so.6(abort+0x143)[0xf75c1c03] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75ab3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3177, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf772f400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf772f425] /lib/libc.so.6(gsignal+0x4f)[0xf756931f] /lib/libc.so.6(abort+0x143)[0xf756ac03] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/cognac/bin/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75543d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3177, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Dec 2013 17:09:13	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	570,240	1,031,716	1.8093
29 Nov 2013 17:30:26	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	544,320	985,466	1.8105
29 Nov 2013 04:33:25	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	518,400	939,489	1.8123
28 Nov 2013 15:50:03	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	492,480	893,767	1.8148
28 Nov 2013 03:00:53	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	466,560	848,014	1.8176
17 Oct 2013 11:19:48	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	440,640	799,876	1.8153
16 Oct 2013 21:37:46	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	414,720	750,930	1.8107
16 Oct 2013 08:04:18	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	388,800	702,770	1.8075
15 Oct 2013 18:39:41	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	362,880	654,879	1.8047
15 Oct 2013 04:53:45	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	336,960	605,814	1.7979
14 Oct 2013 15:33:41	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	311,040	555,724	1.7867
10 Jul 2013 11:06:22	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	285,120	509,058	1.7854
06 Jul 2013 14:23:59	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	259,200	462,391	1.7839
06 Jul 2013 04:50:05	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	233,280	416,111	1.7837
06 Jul 2013 04:07:10	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	207,360	370,364	1.7861
26 Jun 2013 13:01:36	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	181,440	324,038	1.7859
25 Jun 2013 23:58:58	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	155,520	277,440	1.7840
07 Jun 2013 02:07:38	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	129,600	230,885	1.7815
06 Jun 2013 14:10:24	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	103,680	184,595	1.7804
06 Jun 2013 00:12:29	1269200	15828401	hadcm3n_zjn9_1960_40_008389680_3	77,760	138,310	1.7787