<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:2111:2111] [20240619:224037:1718829637:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2111:2111] [20240619:224037:1718829637:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2111:2111] [20240619:224037:1718829637:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2111:2111] [20240619:224037:1718829637:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:41:02 STEP 1824 H= 456:00 +CPU= 20.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:42:19 STEP 1825 H= 456:15 +CPU= 22.081
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1mf
(argv4) batchid: 1018
(argv5) wuid: 12290430
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5737e1c910ab995af33c538e51a1548e to: /var/lib/boinc-client/slots/0/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430.zip
ic_ancil_file: ic_ancil_12290430
ifsdata_file: ifsdata_12290430
climate_data_file: clim_data_12290430
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/0/ic_ancil_12290430.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12290430.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cac4e931df80847f13315dd51f6a1260 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12290430.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12290430.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12290430.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12290430.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12290430.xml
-- Model is restarting --
Adjusting last_iter, 1826, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
last_cpu_time: 42333
upload_file_number: 19
last_iter: 1825
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1mf_2016092300_20_1018_12290430_0_r1790406125
The child process has been launched with process id: 4544
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:224331:1718829811:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:43:54 STEP 1824 H= 456:00 +CPU= 20.395
22:44:15 STEP 1825 H= 456:15 +CPU= 20.398
22:44:36 STEP 1826 H= 456:30 +CPU= 20.569
22:44:57 STEP 1827 H= 456:45 +CPU= 20.130
22:45:19 STEP 1828 H= 457:00 +CPU= 21.156
22:45:41 STEP 1829 H= 457:15 +CPU= 20.631
22:46:02 STEP 1830 H= 457:30 +CPU= 20.275
22:46:22 STEP 1831 H= 457:45 +CPU= 20.182
22:46:44 STEP 1832 H= 458:00 +CPU= 20.712
22:47:05 STEP 1833 H= 458:15 +CPU= 20.032
22:47:26 STEP 1834 H= 458:30 +CPU= 20.657
22:47:46 STEP 1835 H= 458:45 +CPU= 19.710
22:48:11 STEP 1836 H= 459:00 +CPU= 24.395
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001836
22:48:31 STEP 1837 H= 459:15 +CPU= 19.802
22:48:52 STEP 1838 H= 459:30 +CPU= 19.801
22:49:12 STEP 1839 H= 459:45 +CPU= 20.090
22:49:34 STEP 1840 H= 460:00 +CPU= 20.583
22:49:56 STEP 1841 H= 460:15 +CPU= 21.328
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:51:06 STEP 1842 H= 460:30 +CPU= 22.871
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:51:39 STEP 1843 H= 460:45 +CPU= 20.442
22:52:00 STEP 1844 H= 461:00 +CPU= 20.543
double free or corruption (top)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.471] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3536MB (heap), 4796MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.471] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3536 MB (maxheap), 4796 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CPG_DRV
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : >OMP-CPG (1025)
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CPG
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CPG_GP
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GPXX
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225203:1718830323:512.472] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : VERDER
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225204:1718830324:513.472] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:4544:4544] [20240619:225204:1718830324:513.472] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CF935B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008214C1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DD5490 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001783F3F Unknown Unknown Unknown
oifs_43r3_model.e 0000000000C7047C verder_ 106 verder.F90
oifs_43r3_model.e 0000000000D25971 gpxx_ 162 gpxx.F90
oifs_43r3_model.e 0000000000D18EA1 cpg_gp_ 923 cpg_gp.F90
oifs_43r3_model.e 0000000000C2432E cpg_ 627 cpg.F90
oifs_43r3_model.e 0000000000C22238 cpg_drv_ 351 cpg_drv.F90
oifs_43r3_model.e 0000000000BCC914 gp_model_ 419 gp_model.F90
oifs_43r3_model.e 00000000012C5002 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BACBAE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 0000000000688153 stepo_ 327 stepo.F90
oifs_43r3_model.e 00000000004CD1DC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000004B5D85 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 00000000004B4F52 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 000000000049D638 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 0000000000406FBD cnt0_ 146 cnt0.F90
oifs_43r3_model.e 0000000000401E7F MAIN__ 96 master.F90
oifs_43r3_model.e 0000000000401E12 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCC34A Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCDBE7 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000401CD5 Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1mf
(argv4) batchid: 1018
(argv5) wuid: 12290430
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5737e1c910ab995af33c538e51a1548e to: /var/lib/boinc-client/slots/0/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430.zip
ic_ancil_file: ic_ancil_12290430
ifsdata_file: ifsdata_12290430
climate_data_file: clim_data_12290430
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/0/ic_ancil_12290430.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12290430.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cac4e931df80847f13315dd51f6a1260 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12290430.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12290430.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12290430.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12290430.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12290430.xml
-- Model is restarting --
Adjusting last_iter, 1844, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
last_cpu_time: 42763
upload_file_number: 19
last_iter: 1825
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1mf_2016092300_20_1018_12290430_0_r1790406125
The child process has been launched with process id: 4681
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:4681:4681] [20240623:115823:1719136703:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:00:36 STEP 1824 H= 456:00 +CPU= 22.407
12:00:58 STEP 1825 H= 456:15 +CPU= 20.639
12:01:20 STEP 1826 H= 456:30 +CPU= 20.978
12:01:42 STEP 1827 H= 456:45 +CPU= 20.690
12:02:04 STEP 1828 H= 457:00 +CPU= 21.453
12:02:26 STEP 1829 H= 457:15 +CPU= 20.513
12:02:47 STEP 1830 H= 457:30 +CPU= 20.571
12:03:08 STEP 1831 H= 457:45 +CPU= 20.168
12:03:29 STEP 1832 H= 458:00 +CPU= 20.499
12:03:50 STEP 1833 H= 458:15 +CPU= 20.034
12:04:11 STEP 1834 H= 458:30 +CPU= 20.059
12:04:32 STEP 1835 H= 458:45 +CPU= 20.838
12:05:01 STEP 1836 H= 459:00 +CPU= 26.744
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001836
12:05:24 STEP 1837 H= 459:15 +CPU= 21.525
12:05:47 STEP 1838 H= 459:30 +CPU= 21.340
12:06:10 STEP 1839 H= 459:45 +CPU= 21.524
12:06:33 STEP 1840 H= 460:00 +CPU= 21.974
12:06:55 STEP 1841 H= 460:15 +CPU= 21.246
12:07:19 STEP 1842 H= 460:30 +CPU= 21.770
12:07:42 STEP 1843 H= 460:45 +CPU= 22.020
12:08:06 STEP 1844 H= 461:00 +CPU= 21.860
12:08:29 STEP 1845 H= 461:15 +CPU= 21.517
12:08:52 STEP 1846 H= 461:30 +CPU= 21.688
12:09:16 STEP 1847 H= 461:45 +CPU= 22.058
12:09:45 STEP 1848 H= 462:00 +CPU= 26.746
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:19:52 STEP 1849 H= 462:15 +CPU= 22.616
12:20:13 STEP 1850 H= 462:30 +CPU= 20.379
12:20:34 STEP 1851 H= 462:45 +CPU= 20.088
12:20:56 STEP 1852 H= 463:00 +CPU= 20.962
12:21:16 STEP 1853 H= 463:15 +CPU= 19.796
12:21:37 STEP 1854 H= 463:30 +CPU= 20.278
12:21:58 STEP 1855 H= 463:45 +CPU= 20.259
12:22:20 STEP 1856 H= 464:00 +CPU= 21.223
12:22:41 STEP 1857 H= 464:15 +CPU= 20.060
12:23:02 STEP 1858 H= 464:30 +CPU= 20.119
12:23:22 STEP 1859 H= 464:45 +CPU= 20.148
12:23:48 STEP 1860 H= 465:00 +CPU= 24.530
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001860
12:24:07 STEP 1861 H= 465:15 +CPU= 19.592
12:24:27 STEP 1862 H= 465:30 +CPU= 19.241
12:24:46 STEP 1863 H= 465:45 +CPU= 19.245
12:25:06 STEP 1864 H= 466:00 +CPU= 19.710
12:25:26 STEP 1865 H= 466:15 +CPU= 19.232
12:25:46 STEP 1866 H= 466:30 +CPU= 19.482
12:26:07 STEP 1867 H= 466:45 +CPU= 20.200
12:26:28 STEP 1868 H= 467:00 +CPU= 20.429
12:26:48 STEP 1869 H= 467:15 +CPU= 20.138
12:27:09 STEP 1870 H= 467:30 +CPU= 20.081
12:27:30 STEP 1871 H= 467:45 +CPU= 20.052
12:27:58 STEP 1872 H= 468:00 +CPU= 27.021
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001872
12:28:19 STEP 1873 H= 468:15 +CPU= 20.288
12:28:39 STEP 1874 H= 468:30 +CPU= 20.022
12:29:00 STEP 1875 H= 468:45 +CPU= 20.262
12:29:21 STEP 1876 H= 469:00 +CPU= 20.668
12:29:42 STEP 1877 H= 469:15 +CPU= 20.340
12:30:04 STEP 1878 H= 469:30 +CPU= 20.832
12:30:25 STEP 1879 H= 469:45 +CPU= 20.596
12:30:47 STEP 1880 H= 470:00 +CPU= 20.514
12:31:07 STEP 1881 H= 470:15 +CPU= 20.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:31:42 STEP 1882 H= 470:30 +CPU= 22.508
12:32:04 STEP 1883 H= 470:45 +CPU= 21.080
12:32:29 STEP 1884 H= 471:00 +CPU= 24.343
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001884
12:32:50 STEP 1885 H= 471:15 +CPU= 20.245
12:33:10 STEP 1886 H= 471:30 +CPU= 20.100
12:33:31 STEP 1887 H= 471:45 +CPU= 20.282
12:33:52 STEP 1888 H= 472:00 +CPU= 20.340
12:34:14 STEP 1889 H= 472:15 +CPU= 20.766
12:34:35 STEP 1890 H= 472:30 +CPU= 20.193
12:34:56 STEP 1891 H= 472:45 +CPU= 20.335
12:35:17 STEP 1892 H= 473:00 +CPU= 20.326
12:35:37 STEP 1893 H= 473:15 +CPU= 19.842
12:35:57 STEP 1894 H= 473:30 +CPU= 20.041
12:36:19 STEP 1895 H= 473:45 +CPU= 20.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:36:55 STEP 1896 H= 474:00 +CPU= 24.564
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001896
12:37:15 STEP 1897 H= 474:15 +CPU= 19.731
12:37:35 STEP 1898 H= 474:30 +CPU= 19.699
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01
AVE 0.338650869367868E-02 0.999999999999995E-08 0.364610280096703E-01
GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM
AVE 0.186423589056940E-05 0.000000000000000E+00 0.661318045763992E-03
GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM
AVE 0.392424546190006E-06 0.000000000000000E+00 0.906639187083673E-03
GPNORM SNOW AVERAGE MINIMUM MAXIMUM
AVE 0.663335694862293E-06 0.000000000000000E+00 0.309463122186451E-02
GPNORM RAIN AVERAGE MINIMUM MAXIMUM
AVE 0.928357396411514E-07 0.000000000000000E+00 0.719257374362957E-03
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.154383541291195E+00 0.000000000000000E+00 0.100000000000000E+01
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.186301256740798E-05 0.000000000000000E+00 0.661795274738731E-03
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.392921001764827E-06 0.000000000000000E+00 0.892998563719006E-03
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.154367031370378E+00 0.000000000000000E+00 0.100000000000000E+01
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.391545549735335E-06 0.000000000000000E+00 0.308324231638051E-02
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.636524182289934E-06 0.000000000000000E+00 0.179209273919455E-02
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.646353841416335E-06 0.000000000000000E+00 0.592159172074252E-02
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.185227802261139E-06 0.000000000000000E+00 0.442736823099142E-02
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.812665110669751E-01 0.000000000000000E+00 0.100000000000000E+01
NSTEP = 1896 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 1896 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+001896 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+001896
MPI-TASK: 1 - 307445412 BYTES IN 6 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0082( 2.4%)
NSTEP = 1896 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+001896 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+001896
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0066 SECONDS107181.7 Mbytes/s, TOTAL TIME= 0.1763( 3.7%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+001896 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+001896
MPI-TASK: 1 - 4919126592 BYTES IN 125 RECORDS TRANSFERRED IN 0.0046 SECONDS******** Mbytes/s, TOTAL TIME= 0.1509( 3.0%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+001896 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+001896
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0028( 3.6%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+001896 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+001896
MPI-TASK: 1 - 298865683 BYTES IN 4 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0048( 4.2%)
STEPO_FPOS WAS : T00000000M
NSTEP = 1896 STEPO 0AAA00AAA
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 1896 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+001896 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+001896
MPI-TASK: 1 - 307445412 BYTES IN 7 RECORDS TRANSFERRED IN 0.0003 SECONDS******** Mbytes/s, TOTAL TIME= 0.0076( 3.9%)
12:36:55 STEP 1896 H= 474:00 +CPU= 24.564
NSTEP = 1897 STEPO 0AAA00AAA
12:37:15 STEP 1897 H= 474:15 +CPU= 19.731
NSTEP = 1898 STEPO 0AAA00AAA
12:37:35 STEP 1898 H= 474:30 +CPU= 19.699
NSTEP = 1899 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
12:36:20 FULLPOS-B DYNFPOS 1896 1.119 1.119 1.165 25:15 37:57 0.83970112482911E-05 2GB 0MB
12:36:33 FULLPOS-V DYNFPOS 1896 2.286 2.286 13.340 25:17 38:10 0.83970112482911E-05 2GB 0MB
12:36:34 FULLPOS-T DYNFPOS 1896 0.379 0.379 0.383 25:17 38:11 0.83970112482911E-05 2GB 0MB
12:36:34 0AAA00AAA STEPO 1896 0.371 0.371 0.373 25:18 38:11 0.83970112482911E-05 2GB 0MB
12:36:55 0AAA00AAA STEPO 1897 20.277 20.277 20.731 25:38 38:32 0.83970112482911E-05 2GB 0MB
12:37:15 0AAA00AAA STEPO 1898 19.731 19.731 20.207 25:58 38:52 0.83970112482911E-05 2GB 0MB
12:37:35 0AAA00AAA STEPO 1899 19.699 19.699 20.169 26:18 39:12 0.83970112482911E-05 2GB 0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf
CSTEP = " 1872",
CTIME = "00191200 ",
NSTEPLPP = 201, 202, 49, 228226, 228227, 2*0, 1860, 2*0,
IPRGPNSRES = 1,
IPRGPEWRES = 1,
IPRTRWRES = 1,
IPRTRVRES = 1,
GMASS0 = 0.000000000000000E+000,
GMASSI = 0.000000000000000E+000
/
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
<running_values>
<last_cpu_time>44347.930000</last_cpu_time>
<upload_file_number>19</upload_file_number>
<last_iter>1899</last_iter>
<last_upload>1641600</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1mf
(argv4) batchid: 1018
(argv5) wuid: 12290430
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5737e1c910ab995af33c538e51a1548e to: /var/lib/boinc-client/slots/0/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430.zip
ic_ancil_file: ic_ancil_12290430
ifsdata_file: ifsdata_12290430
climate_data_file: clim_data_12290430
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/0/ic_ancil_12290430.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12290430.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cac4e931df80847f13315dd51f6a1260 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12290430.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12290430.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12290430.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12290430.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12290430.xml
-- Model is restarting --
Adjusting last_iter, 1899, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12290430.xml
last_cpu_time: 44347
upload_file_number: 19
last_iter: 1873
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1mf_2016092300_20_1018_12290430_0_r1790406125
The child process has been launched with process id: 12396
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:12396:12396] [20240625:162356:1719325436:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
16:24:20 STEP 1872 H= 468:00 +CPU= 19.965
16:24:40 STEP 1873 H= 468:15 +CPU= 19.229
16:24:59 STEP 1874 H= 468:30 +CPU= 19.195
16:25:19 STEP 1875 H= 468:45 +CPU= 19.350
16:25:39 STEP 1876 H= 469:00 +CPU= 19.685
16:25:59 STEP 1877 H= 469:15 +CPU= 19.345
16:26:18 STEP 1878 H= 469:30 +CPU= 19.408
16:26:38 STEP 1879 H= 469:45 +CPU= 19.290
16:26:58 STEP 1880 H= 470:00 +CPU= 19.573
16:27:17 STEP 1881 H= 470:15 +CPU= 19.247
16:27:37 STEP 1882 H= 470:30 +CPU= 19.279
16:27:56 STEP 1883 H= 470:45 +CPU= 19.301
16:28:20 STEP 1884 H= 471:00 +CPU= 23.797
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001884
16:28:40 STEP 1885 H= 471:15 +CPU= 19.307
16:29:00 STEP 1886 H= 471:30 +CPU= 19.294
16:29:19 STEP 1887 H= 471:45 +CPU= 19.299
16:29:39 STEP 1888 H= 472:00 +CPU= 19.589
16:29:59 STEP 1889 H= 472:15 +CPU= 19.335
16:30:18 STEP 1890 H= 472:30 +CPU= 19.418
16:30:38 STEP 1891 H= 472:45 +CPU= 19.287
16:30:58 STEP 1892 H= 473:00 +CPU= 19.631
16:31:17 STEP 1893 H= 473:15 +CPU= 19.262
16:31:37 STEP 1894 H= 473:30 +CPU= 19.305
16:31:57 STEP 1895 H= 473:45 +CPU= 19.278
16:32:21 STEP 1896 H= 474:00 +CPU= 23.837
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001896
16:32:40 STEP 1897 H= 474:15 +CPU= 19.234
16:33:00 STEP 1898 H= 474:30 +CPU= 19.302
16:33:19 STEP 1899 H= 474:45 +CPU= 19.299
16:33:39 STEP 1900 H= 475:00 +CPU= 19.640
16:33:59 STEP 1901 H= 475:15 +CPU= 19.270
16:34:18 STEP 1902 H= 475:30 +CPU= 19.322
16:34:38 STEP 1903 H= 475:45 +CPU= 19.327
16:34:58 STEP 1904 H= 476:00 +CPU= 19.633
16:35:18 STEP 1905 H= 476:15 +CPU= 19.250
16:35:37 STEP 1906 H= 476:30 +CPU= 19.262
16:35:57 STEP 1907 H= 476:45 +CPU= 19.323
16:36:21 STEP 1908 H= 477:00 +CPU= 23.888
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001908
16:36:40 STEP 1909 H= 477:15 +CPU= 19.266
16:37:00 STEP 1910 H= 477:30 +CPU= 19.326
16:37:20 STEP 1911 H= 477:45 +CPU= 19.345
16:37:40 STEP 1912 H= 478:00 +CPU= 19.692
16:37:59 STEP 1913 H= 478:15 +CPU= 19.263
16:38:19 STEP 1914 H= 478:30 +CPU= 19.324
16:38:38 STEP 1915 H= 478:45 +CPU= 19.337
16:38:58 STEP 1916 H= 479:00 +CPU= 19.651
16:39:18 STEP 1917 H= 479:15 +CPU= 19.340
16:39:38 STEP 1918 H= 479:30 +CPU= 19.278
16:39:57 STEP 1919 H= 479:45 +CPU= 19.299
16:40:22 STEP 1920 H= 480:00 +CPU= 24.553
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01
DDH------------------------------------------------------------------------
I I
I TIME = 480.000H DDH * Diagnostic Var., AREA GLOBAL I
I I
I-------------------------------------------------------------------------I
I VARIABLE I Initial value I Current value I
I-------------------------------------------------------------------------I
I OMEGA (P/S) I -0.2566787946E-21 I 0.6959357631E-07 I
ICloud fractionI 0.0000000000E+00 I 0.5576260357E-01 I
I Relat. Hum. I 0.4061384362E+00 I 0.4444446820E+00 I
I PASS 01 I 0.0000000000E+00 I 0.8741942494E+03 I
DDH------------------------------------------------------------------------
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 1920 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+001920 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+001920
MPI-TASK: 1 - 325488292 BYTES IN 7 RECORDS TRANSFERRED IN 0.0003 SECONDS******** Mbytes/s, TOTAL TIME= 0.0075( 4.0%)
16:40:22 STEP 1920 H= 480:00 +CPU= 24.553
END CNT3
NSTEP = 1920 CNT0 000000000 18.437 18.437 18.753 0.983 0 0 0 0 0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ 46/ 1069125708 WRITTEN 1430/************
-TASK-OPENED-OPEN-RECS IN -KBYTE IN -RECS OUT-KBYTE OUT-WALL -WALL IN-WALL
OU-TOT IN -TOT OUT
1 28 0 46 1044068 1430 ************ 4.4 2.9 1.5 3.4 3.4
===-=== START OF TIMING STATISTICS ===-===
STATS FOR ALL TASKS
NUM ROUTINE CALLS MEAN(ms) MAX(ms) FRAC(%) UNBAL(%)
0 CNT0 - COMPLETE EXECUTION 1 986300.9 986300.9 100.00 0.00
1 CNT4 - FORWARD INTEGRATION 1 982061.6 982061.6 99.57 0.00
8 SCAN2M - GRID-POINT DYNAMICS 53 2881.7 2881.7 15.49 0.00
9 SPCM - SPECTRAL COMP. 48 564.9 564.9 2.75 0.00
10 SCAN2M - PHYSICS 49 10012.6 10012.6 49.74 0.00
11 IOPACK - OUTPUT P.P. RESULTS 4 514.7 514.7 0.21 0.00
12 SPNORM - SPECTRAL NORM COMP. 3 23.5 23.5 0.01 0.00
17 GRIDFPOS IN CNT4 4 16.1 16.1 0.01 0.00
27 RESTART FILES - WRITING 1 2058.8 2058.8 0.21 0.00
28 RESTART FILES - READING 1 3866.0 3866.0 0.39 0.00
29 SU4FPOS IN CNT4 4 0.0 0.0 0.00 0.00
30 DYNFPOS IN CNT4 4 4010.6 4010.6 1.63 0.00
31 POSDDH IN STEPO 1 0.6 0.6 0.00 0.00
37 CPGLAG - SL COMPUTATIONS 49 4116.6 4116.6 20.45 0.00
39 SU0YOMB 1 245.4 245.4 0.02 0.00
51 SCAN2M - SL COMM. PART 1 49 23.9 23.9 0.12 0.00
54 SPCM - M TO S/S TO M TRANSP. 48 261.3 261.3 1.27 0.00
55 SPCIMPF - S TO M/M TO S TRANSP. 48 79.7 79.7 0.39 0.00
56 SPNORM - SPECTRAL NORM COMM. 3 0.1 0.1 0.00 0.00
102 LTINV_CTL - INVERSE LEGENDRE TRANSFORM 108 277.3 277.3 3.04 0.00
103 LTDIR_CTL - DIRECT LEGENDRE TRANSFORM 100 162.9 162.9 1.65 0.00
106 FTDIR_CTL - DIRECT FOURIER TRANSFORM 100 58.7 58.7 0.60 0.00
107 FTINV_CTL - INVERSE FOURIER TRANSFORM 108 101.8 101.8 1.11 0.00
140 SULEG - COMP. OF LEGENDRE POL. 1 30.8 30.8 0.00 0.00
152 LTINV_CTL - M TO L TRANSPOSITION 108 37.9 37.9 0.41 0.00
153 LTDIR_CTL - L TO M TRANSPOSITION 100 20.6 20.6 0.21 0.00
157 FTINV_CTL - L TO G TRANSPOSITION 108 48.9 48.9 0.54 0.00
158 FTDIR_CTL - G TO L TRANSPOSITION 100 15.0 15.0 0.15 0.00
400 GSTATS 6667 0.0 0.0 0.00 0.00
401 GSTATS HOOK 6149 0.0 0.0 0.00 0.00
TOTAL MEASURED IMBALANCE = 0.0 SECONDS, 0.0 PERCENT
TOTAL WALLCLOCK TIME 986.3 CPU TIME 970.5 VECTOR TIME 970.5
===-=== END OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY 1759.6
*** END CNT0 ***
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/0/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/0/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290430/ICMGGh7zg+001824
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1mf_2016092300_20_1018_12290430_0_r1790406125_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
16:43:52 (12392): called boinc_finish(0)
</stderr_txt>
]]>
