climateprediction.net home page
Task 18687039

Task 18687039

Name hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1
Workunit 10008996
Created 8 Jul 2015, 2:43:54 UTC
Sent 27 Oct 2015, 7:49:16 UTC
Report deadline 8 Oct 2016, 13:09:16 UTC
Received 14 Jan 2016, 4:29:00 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1325379
Run time 7 days 20 hours 30 min 38 sec
CPU time 7 days 9 hours 54 min 45 sec
Validate state Workunit error - check skipped
Credit 9,559.58
Device peak FLOPS 2.42 GFLOPS
Application version UK Met Office HadAM3P and HadRM3P model with MOSES II and TRIFFID Europe v7.01
i686-pc-linux-gnu
Stderr
<core_client_version>7.2.42</core_client_version>
<![CDATA[
<stderr_txt>
m3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf72f8545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2369, selfPID=2149, iMonCtr=1
SIGSEGV: segmentation violation
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (9 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77d3b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x3b0)[0x804d524]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80508af]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f63]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73e9545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf775fb70]
/lib/libc.so.6(getenv+0x9a)[0xf738d2ca]
/lib/libc.so.6(+0xaaadb)[0xf7407adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7405a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf775fb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf775fb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf775fba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7415e56]
/lib/libc.so.6(usleep+0x35)[0xf744b5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7375545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
07:01:03 (2392): called boinc_finish
07:01:03 (2393): called boinc_finish
07:01:03 (2392): called boinc_finish
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2392, selfPID=2393, iMonCtr=1
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7749b70]
/lib/libc.so.6(getenv+0x9a)[0xf73772ca]
/lib/libc.so.6(+0xaaadb)[0xf73f1adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73efa2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7749b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7749b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7749ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73ffe56]
/lib/libc.so.6(usleep+0x35)[0xf74355b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf735f545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e9b70]
/lib/libc.so.6(getenv+0x9a)[0xf73172ca]
/lib/libc.so.6(+0xaaadb)[0xf7391adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf738fa2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e9b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76e9ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf76afa0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804feba]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e9b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76e9ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf739fe56]
/lib/libc.so.6(usleep+0x35)[0xf73d55b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf72ff545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2570, selfPID=1971, iMonCtr=1
Model crash detected, will try to restart...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76deb70]
/lib/libc.so.6(getenv+0x9a)[0xf730c2ca]
/lib/libc.so.6(+0xaaadb)[0xf7386adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7384a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76deb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76deb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76deba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7394e56]
/lib/libc.so.6(usleep+0x35)[0xf73ca5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf769b452]
/lib/libc.so.6(clone+0x5e)[0xf73d42fe]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a0b70]
/lib/libc.so.6(getenv+0x9a)[0xf73ce2ca]
/lib/libc.so.6(+0xaaadb)[0xf7448adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7446a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a0b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a0b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77a0ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7456e56]
/lib/libc.so.6(usleep+0x35)[0xf748c5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf775d452]
/lib/libc.so.6(clone+0x5e)[0xf74962fe]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7772b70]
/lib/libc.so.6(getenv+0x9a)[0xf73a02ca]
/lib/libc.so.6(+0xaaadb)[0xf741aadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7418a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7772b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7772ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf7738a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7772b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7772ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7428e56]
/lib/libc.so.6(usleep+0x35)[0xf745e5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7388545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
07:01:04 (2664): called boinc_finish
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77d2b70]
/lib/libc.so.6(getenv+0x9a)[0xf74002ca]
/lib/libc.so.6(+0xaaadb)[0xf747aadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7478a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77d2b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77d2b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77d2ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7488e56]
/lib/libc.so.6(usleep+0x35)[0xf74be5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf778f452]
/lib/libc.so.6(clone+0x5e)[0xf74c82fe]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7780b70]
/lib/libc.so.6(getenv+0x9a)[0xf73ae2ca]
/lib/libc.so.6(+0xaaadb)[0xf7428adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7426a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7780b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7780b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7780ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7436e56]
/lib/libc.so.6(usleep+0x35)[0xf746c5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf773d452]
/lib/libc.so.6(clone+0x5e)[0xf74762fe]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a7b70]
/lib/libc.so.6(getenv+0x9a)[0xf73d52ca]
/lib/libc.so.6(+0xaaadb)[0xf744fadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf744da2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a7b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a7b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77a7ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf745de56]
/lib/libc.so.6(usleep+0x35)[0xf74935b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73bd545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
SIGSEGV: segmentation violation
Signal 15 received, exiting...
07:01:03 (2757): called boinc_finish
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fbb70]
/lib/libc.so.6(getenv+0x9a)[0xf73292ca]
/lib/libc.so.6(+0xaaadb)[0xf73a3adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73a1a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fbb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fbb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76fbba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73b1e56]
/lib/libc.so.6(usleep+0x35)[0xf73e75b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7311545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7763b70]
/lib/libc.so.6(getenv+0x9a)[0xf73912ca]
/lib/libc.so.6(+0xaaadb)[0xf740badb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7409a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7763b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7763ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf7729a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7763b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7763ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7419e56]
/lib/libc.so.6(usleep+0x35)[0xf744f5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7379545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f3b70]
/lib/libc.so.6(getenv+0x9a)[0xf73212ca]
/lib/libc.so.6(+0xaaadb)[0xf739badb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7399a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f3b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76f3ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf76b9a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804feba]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f3b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76f3ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73a9e56]
/lib/libc.so.6(usleep+0x35)[0xf73df5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7309545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2522, selfPID=1970, iMonCtr=1
Model crash detected, will try to restart...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771bb70]
/lib/libc.so.6(getenv+0x9a)[0xf73492ca]
/lib/libc.so.6(+0xaaadb)[0xf73c3adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73c1a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771bb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771bb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf771bba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73d1e56]
/lib/libc.so.6(usleep+0x35)[0xf74075b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf76d8452]
/lib/libc.so.6(clone+0x5e)[0xf74112fe]

Exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
/lib/libc.so.6(getenv+0x9a)[0xf73c92ca]
/lib/libc.so.6(+0xaaadb)[0xf7443adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7441a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf779bba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7451e56]
/lib/libc.so.6(usleep+0x35)[0xf74875b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73b1545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (21 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e8b70]
/lib/libc.so.6(getenv+0x9a)[0xf73162ca]
/lib/libc.so.6(+0xaaadb)[0xf7390adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf738ea2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e8b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c46a0]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e8b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76e8b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76e8ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf739ee56]
/lib/libc.so.6(usleep+0x35)[0xf73d45b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf72fe545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf770eb70]
/lib/libc.so.6(getenv+0x9a)[0xf733c2ca]
/lib/libc.so.6(+0xaaadb)[0xf73b6adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73b4a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf770eb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf770eb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf770eba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73c4e56]
/lib/libc.so.6(usleep+0x35)[0xf73fa5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7324545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7790b70]
/lib/libc.so.6(getenv+0x9a)[0xf73be2ca]
/lib/libc.so.6(+0xaaadb)[0xf7438adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7436a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7790b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7790ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf7756a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7790b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7790ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7446e56]
/lib/libc.so.6(usleep+0x35)[0xf747c5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73a6545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c2b70]
/lib/libc.so.6(getenv+0x9a)[0xf73f02ca]
/lib/libc.so.6(+0xaaadb)[0xf746aadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7468a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c2b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77c2b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77c2ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7478e56]
/lib/libc.so.6(usleep+0x35)[0xf74ae5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73d8545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7785b70]
/lib/libc.so.6(getenv+0x9a)[0xf73b32ca]
/lib/libc.so.6(+0xaaadb)[0xf742dadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf742ba2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7785b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7785ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf774ba0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7785b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7785ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf743be56]
/lib/libc.so.6(usleep+0x35)[0xf74715b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf739b545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7707b70]
/lib/libc.so.6(getenv+0x9a)[0xf73352ca]
/lib/libc.so.6(+0xaaadb)[0xf73afadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73ada2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7707b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7707b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7707ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73bde56]
/lib/libc.so.6(usleep+0x35)[0xf73f35b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf731d545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2417, selfPID=2041, iMonCtr=1
SIGSEGV: segmentation violation
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (9 frames):
Stack trace (../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
17linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7782b70]
 frames):../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x3b0)[0x804d524]

../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80508af]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f63]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7782b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
/lib/libc.so.6(getenv+0x9a)[0xf73b02ca]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(+0xaaadb)[0xf742aadb]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7398545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fbb70]
/lib/libc.so.6(getenv+0x9a)[0xf73292ca]
/lib/libc.so.6(+0xaaadb)[0xf73a3adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73a1a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fbb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fbb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76fbba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73b1e56]
/lib/libc.so.6(usleep+0x35)[0xf73e75b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7311545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7732b70]
/lib/libc.so.6(getenv+0x9a)[0xf73602ca]
/lib/libc.so.6(+0xaaadb)[0xf73daadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73d8a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7732b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7732b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7732ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73e8e56]
/lib/libc.so.6(usleep+0x35)[0xf741e5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7348545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
15:19:39 (2469): called boinc_finish
Signal 1 received, exiting...
15:19:39 (1941): called boinc_finish
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773ab70]
/lib/libc.so.6(getenv+0x9a)[0xf73682ca]
/lib/libc.so.6(+0xaaadb)[0xf73e2adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73e0a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773ab70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773ab70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf773aba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73f0e56]
/lib/libc.so.6(usleep+0x35)[0xf74265b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7350545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2582, selfPID=2117, iMonCtr=1
Model crash detected, will try to restart...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fdb70]
/lib/libc.so.6(getenv+0x9a)[0xf732b2ca]
/lib/libc.so.6(+0xaaadb)[0xf73a5adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73a3a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fdb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76fdb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76fdba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73b3e56]
/lib/libc.so.6(usleep+0x35)[0xf73e95b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf76ba452]
/lib/libc.so.6(clone+0x5e)[0xf73f32fe]

Exiting...
Signal 1 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7761b70]
/lib/libc.so.6(getenv+0x9a)[0xf738f2ca]
/lib/libc.so.6(+0xaaadb)[0xf7409adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7407a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7761b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7761b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7761ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7417e56]
/lib/libc.so.6(usleep+0x35)[0xf744d5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf771e452]
/lib/libc.so.6(clone+0x5e)[0xf74572fe]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
07:01:03 (2634): called boinc_finish
07:01:03 (2634): called boinc_finish
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
/lib/libc.so.6(getenv+0x9a)[0xf73c92ca]
/lib/libc.so.6(+0xaaadb)[0xf7443adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7441a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf779bba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf7761a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf779bba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7451e56]
/lib/libc.so.6(usleep+0x35)[0xf74875b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73b1545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7792b70]
/lib/libc.so.6(getenv+0x9a)[0xf73c02ca]
/lib/libc.so.6(+0xaaadb)[0xf743aadb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7438a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7792b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7792b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7792ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7448e56]
/lib/libc.so.6(usleep+0x35)[0xf747e5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73a8545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cab70]
/lib/libc.so.6(getenv+0x9a)[0xf73f82ca]
/lib/libc.so.6(+0xaaadb)[0xf7472adb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7470a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cab70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cab70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77caba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7480e56]
/lib/libc.so.6(usleep+0x35)[0xf74b65b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf7787452]
/lib/libc.so.6(clone+0x5e)[0xf74c02fe]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7763b70]
/lib/libc.so.6(getenv+0x9a)[0xf73912ca]
/lib/libc.so.6(+0xaaadb)[0xf740badb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7409a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7763b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7763ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf7729a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7763b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7763ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7419e56]
/lib/libc.so.6(usleep+0x35)[0xf744f5b5]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7379545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
14:44:07 (2513): called boinc_finish
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf777bb70]
/lib/libc.so.6(getenv+0x9a)[0xf73a92ca]
/lib/libc.so.6(+0xaaaeb)[0xf7423aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7421a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf777bb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf777bb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf777bba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7431e66]
/lib/libc.so.6(usleep+0x35)[0xf7467515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7391545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7754b70]
/lib/libc.so.6(getenv+0x9a)[0xf73822ca]
/lib/libc.so.6(+0xaaaeb)[0xf73fcaeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73faa2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7754b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7754ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf740ae66]
/lib/libc.so.6(usleep+0x35)[0xf7440515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf736a545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
/lib/libc.so.6(getenv+0x9a)[0xf73c92ca]
/lib/libc.so.6(+0xaaaeb)[0xf7443aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7441a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779bb70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf779bba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7451e66]
/lib/libc.so.6(usleep+0x35)[0xf7487515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73b1545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2468, selfPID=2044, iMonCtr=1
Signal 15 received, exiting...
SIGSEGV: segmentation violation
SIGSEGV: segmentation violation
Stack trace (9Stack trace ( frames):17
 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76ebb70]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76ebb70]
/lib/libc.so.6(getenv+0x9a)[0xf73192ca]
/lib/libc.so.6(+0xaaaeb)[0xf7393aeb]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x3b0)[0x804d524]
/lib/libc.so.6(localtime_r+0x1e)[0xf7391a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80508af]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f63]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76ebb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7301545]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76ebb70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76ebba0]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7778b70]
/lib/libc.so.6(getenv+0x9a)[0xf73a62ca]
/lib/libc.so.6(+0xaaaeb)[0xf7420aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf741ea2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7778b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7778ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf773ea0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7778b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7778ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf742ee66]
/lib/libc.so.6(usleep+0x35)[0xf7464515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf738e545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 15 received, exiting...
Signal 15 received, exiting...
07:01:08 (2184): called boinc_finish
Signal 1 received, exiting...
07:01:08 (2185): called boinc_finish
Signal 15 received, exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f1b70]
/lib/libc.so.6(getenv+0x9a)[0xf731f2ca]
/lib/libc.so.6(+0xaaaeb)[0xf7399aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7397a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f1b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f1b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76f1ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73a7e66]
/lib/libc.so.6(usleep+0x35)[0xf73dd515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7307545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
07:01:04 (2580): called boinc_finish
Signal 15 received, exiting...
SIGSEGV: segmentation violation
SIGSEGV: segmentation violation
Stack trace (17 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771db70]
/lib/libc.so.6(getenv+0x9a)[0xf734b2ca]
/lib/libc.so.6(+0xaaaeb)[0xf73c5aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73c3a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771db70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771db70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf771dba0]
/lib/libc.so.6(nanosleep+0x46)[0xf73d3e66]
/lib/libc.so.6(usleep+0x35)[0xf7409515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
/lib/libpthread.so.0(+0x6452)[0xf76da452]
/lib/libc.so.6(clone+0x5e)[0xf741325e]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7767b70]
/lib/libc.so.6(getenv+0x9a)[0xf73952ca]
/lib/libc.so.6(+0xaaaeb)[0xf740faeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf740da2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7767b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7767ba0]
/lib/libc.so.6(kill+0x16)[0xf7392b66]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe62]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7767b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7767ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf741de66]
/lib/libc.so.6(usleep+0x35)[0xf7453515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf737d545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
SIGSEGV: segmentation violation
Stack trace (Stack trace (717 frames): frames):

../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7775b70]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7775b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f7d]
/lib/libc.so.6(getenv+0x9a)[0xf73a32ca]
/lib/libc.so.6(+0xaaaeb)[0xf741daeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf741ba2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7775b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7775b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7775ba0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(nanosleep+0x46)[0xf742be66]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf738b545]
/lib/libc.so.6(usleep+0x35)[0xf7461515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]

Exiting...
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c5884]
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (24 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b2b70]
/lib/libc.so.6(getenv+0x9a)[0xf73e02ca]
/lib/libc.so.6(+0xaaaeb)[0xf745aaeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf7458a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b2b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77b2ba0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf7778a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804feba]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b2b70]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c46a0]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b2b70]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf77b2ba0]
/lib/libc.so.6(nanosleep+0x46)[0xf7468e66]
/lib/libc.so.6(usleep+0x35)[0xf749e515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf73c8545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
SIGSEGV: segmentation violation
Stack trace (19 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7719bc0]
/lib/libc.so.6(getenv+0x9a)[0xf73472ca]
/lib/libc.so.6(+0xaaaeb)[0xf73c1aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73bfa2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7719bc0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d6ec]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7719bc0]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7719bf0]
/lib/libc.so.6(nanosleep+0x46)[0xf73cfe66]
/lib/libc.so.6(usleep+0x35)[0xf7405515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf732f545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Signal 15 received, exiting...
Signal 1 received, exiting...
SIGSEGV: segmentation violation
Stack trace (22 frames):
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80c8ccf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771ebc0]
/lib/libc.so.6(getenv+0x9a)[0xf734c2ca]
/lib/libc.so.6(+0xaaaeb)[0xf73c6aeb]
/lib/libc.so.6(localtime_r+0x1e)[0xf73c4a2e]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c4306]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80c47b6]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771ebc0]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf771ebf0]
/lib/libpthread.so.0(waitpid+0x4b)[0xf76e4a0b]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804fe77]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x804d720]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771ebc0]
linux-gate.so.1(__kernel_vsyscall+0x10)[0xf771ebf0]
/lib/libc.so.6(nanosleep+0x46)[0xf73d4e66]
/lib/libc.so.6(usleep+0x35)[0xf740a515]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x80da8e0]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8050f74]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051aa7]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu[0x8051d79]
/lib/libc.so.6(__libc_start_main+0xf5)[0xf7334545]
../../projects/climateprediction.net/hadam3prm3pm2t_eu_7.01_i686-pc-linux-gnu(__gxx_personality_v0+0x1bd)[0x804d331]

Exiting...
Leaving CPDN_Main::Monitor...
12:23:05 (1863): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Jan 2016 02:16:48 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 138,539 651,591 4.7033
09 Jan 2016 20:30:19 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 127,019 593,805 4.6749
04 Jan 2016 04:55:07 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 115,499 536,356 4.6438
24 Dec 2015 20:32:35 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 103,979 479,049 4.6072
15 Dec 2015 08:02:44 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 92,459 421,964 4.5638
04 Dec 2015 02:24:24 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 80,939 364,862 4.5079
28 Nov 2015 07:54:22 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 69,419 307,587 4.4309
19 Nov 2015 00:24:58 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 57,899 252,759 4.3655
14 Nov 2015 01:16:23 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 46,379 202,093 4.3574
06 Nov 2015 03:42:19 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 34,859 151,976 4.3597
05 Nov 2015 20:07:32 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 23,339 101,955 4.3684
05 Nov 2015 20:07:32 1325379 18687039 hadam3prm3pm2t_eu_jm1b_2002_1_010009591_1 11,819 52,683 4.4575


©2024 climateprediction.net