Task 15852709

Name	hadcm3n_u3wf_2020_40_008391656_0
Workunit	8542515
Created	21 Jun 2013, 15:35:27 UTC
Sent	1 Jul 2013, 18:22:06 UTC
Report deadline	1 Oct 2013, 1:49:17 UTC
Received	21 Sep 2013, 16:49:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1195874
Run time	54 days 18 hours 29 min 43 sec
CPU time	46 days 12 hours 34 min 50 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	0.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:58:39 (12918): No heartbeat from core client for 30 sec - exiting 07:58:41 (12918): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1527, iMonCtr=1 Model crash detected, will try to restart... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (6 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xcbd400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051147] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x21bbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x165400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x875bd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 22:15:10 (1486): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:49 (2861): No heartbeat from core client for 30 sec - exiting 22:15:50 (2861): No heartbeat from core client for 30 sec - exiting 22:15:51 (2861): No heartbeat from core client for 30 sec - exiting 22:15:52 (2861): No heartbeat from core client for 30 sec - exiting 22:15:53 (2861): No heartbeat from core client for 30 sec - exiting 22:15:54 (2861): No heartbeat from core client for 30 sec - exiting 22:15:56 (2861): No heartbeat from core client for 30 sec - exiting 22:15:57 (2861): No heartbeat from core client for 30 sec - exiting 22:15:58 (2861): No heartbeat from core client for 30 sec - exiting 22:15:59 (2861): No heartbeat from core client for 30 sec - exiting 22:16:00 (2861): No heartbeat from core client for 30 sec - exiting 22:16:01 (2861): No heartbeat from core client for 30 sec - exiting 22:16:02 (2861): No heartbeat from core client for 30 sec - exiting 22:16:03 (2861): No heartbeat from core client for 30 sec - exiting 22:16:04 (2861): No heartbeat from core client for 30 sec - exiting 22:16:05 (2861): No heartbeat from core client for 30 sec - exiting 22:16:06 (2861): No heartbeat from core client for 30 sec - exiting 22:16:07 (2861): No heartbeat from core client for 30 sec - exiting 22:16:08 (2861): No heartbeat from core client for 30 sec - exiting 22:16:09 (2861): No heartbeat from core client for 30 sec - exiting 22:16:10 (2861): No heartbeat from core client for 30 sec - exiting 22:16:11 (2861): No heartbeat from core client for 30 sec - exiting 22:16:12 (2861): No heartbeat from core client for 30 sec - exiting 22:16:13 (2861): No heartbeat from core client for 30 sec - exiting 22:16:14 (2861): No heartbeat from core client for 30 sec - exiting 22:16:15 (2861): No heartbeat from core client for 30 sec - exiting 22:16:16 (2861): No heartbeat from core client for 30 sec - exiting 22:16:17 (2861): No heartbeat from core client for 30 sec - exiting 22:16:18 (2861): No heartbeat from core client for 30 sec - exiting 22:16:19 (2861): No heartbeat from core client for 30 sec - exiting 22:16:20 (2861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:00:21 (1483): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:03 (4657): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:23 (4662): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:03:23 (4666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:04:04 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:05 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:45 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:46 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:07:26 (4689): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:27 (4693): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:17 (4697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:58 (4702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:39 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:39 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish 17:45:19 (1317):Signal 15 received, exiting... No heartbeat from core client for 30 sec - exiting Called boinc_finish 17:45:21 (1317): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 12:38:54 (8694): No heartbeat from core client for 30 sec - exiting Called boinc_finish 12:38:55 (8694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 08:23:00 (1531): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:03 (1531): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish 08:32:03 (1149): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 07:43:20 (2025): No heartbeat from core client for 30 sec - exiting 07:44:15 (2025): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2489, iMonCtr=1 Model crash detected, will try to restart... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (6 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xc38400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051147] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xa84bd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1497, iMonCtr=1 Model crash detected, will try to restart... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (6 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x25c400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051147] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xaecbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6273, iMonCtr=1 Model crash detected, will try to restart... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xc69400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x241bd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x8bc400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805067e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xadfbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... 17:23:41 (1491): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:44 (1491): No heartbeat from core client for 30 sec - exiting 17:23:45 (1491): No heartbeat from core client for 30 sec - exiting 17:24:25 (3853): No heartbeat from core client for 30 sec - exiting 17:24:26 (3853): No heartbeat from core client for 30 sec - exiting 17:24:27 (3853): No heartbeat from core client for 30 sec - exiting 17:24:28 (3853): No heartbeat from core client for 30 sec - exiting 17:24:29 (3853): No heartbeat from core client for 30 sec - exiting 17:24:30 (3853): No heartbeat from core client for 30 sec - exiting 17:24:31 (3853): No heartbeat from core client for 30 sec - exiting 17:24:32 (3853): No heartbeat from core client for 30 sec - exiting 17:24:33 (3853): No heartbeat from core client for 30 sec - exiting 17:24:34 (3853): No heartbeat from core client for 30 sec - exiting 17:24:35 (3853): No heartbeat from core client for 30 sec - exiting 17:24:36 (3853): No heartbeat from core client for 30 sec - exiting 17:24:37 (3853): No heartbeat from core client for 30 sec - exiting 17:24:38 (3853): No heartbeat from core client for 30 sec - exiting 17:24:39 (3853): No heartbeat from core client for 30 sec - exiting 17:24:40 (3853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:29:17 (5922): No heartbeat from core client for 30 sec - exiting 08:30:22 (5922): No heartbeat from core client for 30 sec - exiting 08:30:23 (5922): No heartbeat from core client for 30 sec - exiting 08:30:24 (5922): No heartbeat from core client for 30 sec - exiting 08:30:28 (5922): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:29 (5922): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish 18:44:39 (1499): No heartbeat from core client for 30 sec - exiting 18:44:46 (1499): No heartbeat from core client for 30 sec - exiting 18:44:47 (1499): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:48 (1499): No heartbeat from core client for 30 sec - exiting 18:44:49 (1499): No heartbeat from core client for 30 sec - exiting 18:44:50 (1499): No heartbeat from core client for 30 sec - exiting 18:44:51 (1499): No heartbeat from core client for 30 sec - exiting 18:44:52 (1499): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:39:16 (2401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:25 (6259): No heartbeat from core client for 30 sec - exiting 07:42:07 (6259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:15 (6259): No heartbeat from core client for 30 sec - exiting 07:42:16 (6259): No heartbeat from core client for 30 sec - exiting 07:42:17 (6259): No heartbeat from core client for 30 sec - exiting 07:42:18 (6259): No heartbeat from core client for 30 sec - exiting 07:42:19 (6259): No heartbeat from core client for 30 sec - exiting 07:42:21 (6259): No heartbeat from core client for 30 sec - exiting 07:42:22 (6259): No heartbeat from core client for 30 sec - exiting 07:42:24 (6259): No heartbeat from core client for 30 sec - exiting 08:36:21 (1526): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08681848 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6(+0x6e341)[0x242341] /lib/tls/i686/cmov/libc.so.6(+0x6fb98)[0x243b98] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x246c7d] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xed2741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xed279d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x1eabd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00110000-0012d000 r-xp 00000000 08:01 3407899 /lib/libgcc_s.so.1 0012d000-0012e000 r--p 0001c000 08:01 3407899 /lib/libgcc_s.so.1 0012e000-0012f000 rw-p 0001d000 08:01 3407899 /lib/libgcc_s.so.1 001d4000-0032d000 r-xp 00000000 08:01 3548072 /lib/tls/i686/cmov/libc-2.11.1.so 0032d000-0032f000 r--p 00159000 08:01 3548072 /lib/tls/i686/cmov/libc-2.11.1.so 0032f000-00330000 rw-p 0015b000 08:01 3548072 /lib/tls/i686/cmov/libc-2.11.1.so 00330000-00333000 rw-p 00000000 00:00 0 003e6000-003fb000 r-xp 00000000 08:01 3548062 /lib/tls/i686/cmov/libpthread-2.11.1.so 003fb000-003fc000 r--p 00014000 08:01 3548062 /lib/tls/i686/cmov/libpthread-2.11.1.so 003fc000-003fd000 rw-p 00015000 08:01 3548062 /lib/tls/i686/cmov/libpthread-2.11.1.so 003fd000-003ff000 rw-p 00000000 00:00 0 00568000-0058c000 r-xp 00000000 08:01 3548066 /lib/tls/i686/cmov/libm-2.11.1.so 0058c000-0058d000 r--p 00023000 08:01 3548066 /lib/tls/i686/cmov/libm-2.11.1.so 0058d000-0058e000 rw-p 00024000 08:01 3548066 /lib/tls/i686/cmov/libm-2.11.1.so 005bb000-005bd000 r-xp 00000000 08:01 3548064 /lib/tls/i686/cmov/libdl-2.11.1.so 005bd000-005be000 r--p 00001000 08:01 3548064 /lib/tls/i686/cmov/libdl-2.11.1.so 005be000-005bf000 rw-p 00002000 08:01 3548064 /lib/tls/i686/cmov/libdl-2.11.1.so 006fa000-006fb000 r-xp 00000000 00:00 0 [vdso] 00d96000-00db1000 r-xp 00000000 08:01 3414077 /lib/ld-2.11.1.so 00db1000-00db2000 r--p 0001a000 08:01 3414077 /lib/ld-2.11.1.so 00db2000-00db3000 rw-p 0001b000 08:01 3414077 /lib/ld-2.11.1.so 00e17000-00f00000 r-xp 00000000 08:01 657253 /usr/lib/libstdc++.so.6.0.13 00f00000-00f04000 r--p 000e9000 08:01 657253 /usr/lib/libstdc++.so.6.0.13 00f04000-00f05000 rw-p 000ed000 08:01 657253 /usr/lib/libstdc++.so.6.0.13 00f05000-00f0c000 rw-p 00000000 00:00 0 08048000-080e3000 r-xp 00000000 08:01 4459059 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 4459059 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0862c000-08692000 rw-p 00000000 00:00 0 [heap] b7100000-b7121000 rw-p 00000000 00:00 0 b7121000-b7200000 ---p 00000000 00:00 0 b72b7000-b7732000 rw-s 00000000 08:01 132067 /var/lib/boinc-client/slots/0/137215 b7732000-b7735000 rw-p 00000000 00:00 0 b7741000-b7742000 rw-p 00000000 00:00 0 b7742000-b7743000 ---p 00000000 00:00 0 b7743000-b7746000 rw-p 00000000 00:00 0 b7746000-b7748000 rw-s 00000000 08:01 132063 /var/lib/boinc-client/slots/0/boinc_mmap_file b7748000-b774a000 rw-p 00000000 00:00 0 bfdc6000-bfe35000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x6fa400] [0x6fa422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0x1fe681] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0x201ab2] /lib/tls/i686/cmov/libc.so.6(+0x6424d)[0x23824d] /lib/tls/i686/cmov/libc.so.6(+0x6e341)[0x242341] /lib/tls/i686/cmov/libc.so.6(+0x6fb98)[0x243b98] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x246c7d] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xed2741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xed279d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x1eabd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Sep 2013 16:54:41	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	777,600	4,019,677	5.1693
18 Sep 2013 17:32:11	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	751,680	3,887,455	5.1717
16 Sep 2013 23:05:33	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	725,760	3,757,265	5.1770
15 Sep 2013 06:33:19	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	699,840	3,629,706	5.1865
13 Sep 2013 11:41:44	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	673,920	3,499,072	5.1921
11 Sep 2013 15:17:47	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	648,000	3,367,306	5.1965
09 Sep 2013 20:49:20	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	622,080	3,235,035	5.2004
08 Sep 2013 01:29:27	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	596,160	3,098,241	5.1970
06 Sep 2013 06:21:11	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	570,240	2,961,971	5.1943
03 Sep 2013 15:18:12	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	544,320	2,825,613	5.1911
01 Sep 2013 18:48:30	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	518,400	2,690,082	5.1892
30 Aug 2013 22:34:45	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	492,480	2,553,841	5.1857
29 Aug 2013 02:17:47	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	466,560	2,418,068	5.1828
27 Aug 2013 05:29:59	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	440,640	2,282,298	5.1795
24 Aug 2013 10:58:41	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	414,720	2,146,386	5.1755
22 Aug 2013 15:26:31	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	388,800	2,010,725	5.1716
20 Aug 2013 18:06:00	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	362,880	1,874,813	5.1665
18 Aug 2013 20:32:05	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	336,960	1,740,636	5.1657
17 Aug 2013 01:28:24	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	311,040	1,605,983	5.1633
15 Aug 2013 05:34:26	1195874	15852709	hadcm3n_u3wf_2020_40_008391656_0	285,120	1,469,737	5.1548