<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
a_file: ifsdata_12169300
climate_data_file: clim_data_12169300
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_051dea1693c95b80c1b9121ce69d77fd to: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4046bbe13aea1bbc3b8858012b09cd85 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715
The child process has been launched with process id: 1492289
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015037:1673225437:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
double free or corruption (top)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.737] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2865MB (heap), 2342MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.743] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.743] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2865 MB (maxheap), 2343 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADDRV 2 (1202)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADCFG
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADPAR
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SURFRAD
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015950:1673225990:552.751] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SURFRAD_CTL_MOD:SURFRAD_CTL
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015951:1673225991:553.751] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1492289:1492289] [20230109:015951:1673225991:553.752] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 00000000017B2FB9 surfrad_ctl_mod_m 427 surfrad_ctl_mod.F90
oifs_43r3_model.e 00000000015C72FD surfrad_ 279 surfrad.F90
oifs_43r3_model.e 000000000127E4C6 radpar_ 421 radpar.F90
oifs_43r3_model.e 000000000127B7B0 radcfg_ 306 radcfg.F90
oifs_43r3_model.e 000000000126A40A raddrv_ 413 raddrv.F90
oifs_43r3_model.e 0000000000BFD839 gp_model_ 482 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0656
(argv4) batchid: 952
(argv5) wuid: 12169300
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2686d596a53b64a6b0d7aa9cf38688fa to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0656_1983050100_123_952_12169300.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0656_1983050100_123_952_12169300.zip
ic_ancil_file: ic_ancil_12169300
ifsdata_file: ifsdata_12169300
climate_data_file: clim_data_12169300
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_051dea1693c95b80c1b9121ce69d77fd to: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4046bbe13aea1bbc3b8858012b09cd85 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12169300.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
last_cpu_time: 19
upload_file_number: 0
last_iter: -1
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715
The child process has been launched with process id: 1622550
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233617:1673476577:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
double free or corruption (top)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.458] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3759MB (heap), 3153MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.458] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.458] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.458] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.459] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.459] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3759 MB (maxheap), 3153 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALL_SL
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LAPINEA
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LARMES
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LARCINA
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233658:1673476618:41.461] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LASCAW
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233659:1673476619:42.461] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1622550:1622550] [20230111:233659:1673476619:42.461] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000E01D5A lascaw_ 493 lascaw.F90
oifs_43r3_model.e 0000000000DED525 larcina_ 319 larcina.F90
oifs_43r3_model.e 0000000000DEABF8 larmes_ 603 larmes.F90
oifs_43r3_model.e 0000000000DE50FD lapinea_ 270 lapinea.F90
oifs_43r3_model.e 0000000000D7A4EC call_sl_ 300 call_sl.F90
oifs_43r3_model.e 0000000000D784FD call_sl_heap_ 94 call_sl_heap.F90
oifs_43r3_model.e 0000000000BF9958 gp_model_ 533 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0656
(argv4) batchid: 952
(argv5) wuid: 12169300
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2686d596a53b64a6b0d7aa9cf38688fa to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0656_1983050100_123_952_12169300.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0656_1983050100_123_952_12169300.zip
ic_ancil_file: ic_ancil_12169300
ifsdata_file: ifsdata_12169300
climate_data_file: clim_data_12169300
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_051dea1693c95b80c1b9121ce69d77fd to: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4046bbe13aea1bbc3b8858012b09cd85 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12169300.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
last_cpu_time: 35
upload_file_number: 0
last_iter: -1
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715
The child process has been launched with process id: 1623082
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1623082:1623082] [20230112:000010:1673478010:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
00:00:41 STEP 0 H= 0:00 +CPU= 28.393
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000000
00:00:58 STEP 1 H= 1:00 +CPU= 17.203
00:01:14 STEP 2 H= 2:00 +CPU= 15.899
00:01:46 STEP 3 H= 3:00 +CPU= 30.774
00:02:07 STEP 4 H= 4:00 +CPU= 21.035
00:02:28 STEP 5 H= 5:00 +CPU= 20.659
00:03:01 STEP 6 H= 6:00 +CPU= 32.688
00:03:22 STEP 7 H= 7:00 +CPU= 20.330
00:03:43 STEP 8 H= 8:00 +CPU= 21.120
00:04:16 STEP 9 H= 9:00 +CPU= 32.723
00:04:38 STEP 10 H= 10:00 +CPU= 21.415
00:04:59 STEP 11 H= 11:00 +CPU= 20.784
00:05:36 STEP 12 H= 12:00 +CPU= 36.340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000012
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000012
00:05:57 STEP 13 H= 13:00 +CPU= 20.420
00:06:19 STEP 14 H= 14:00 +CPU= 21.165
00:06:52 STEP 15 H= 15:00 +CPU= 32.482
00:07:14 STEP 16 H= 16:00 +CPU= 21.637
00:07:35 STEP 17 H= 17:00 +CPU= 21.117
00:08:09 STEP 18 H= 18:00 +CPU= 33.299
00:08:30 STEP 19 H= 19:00 +CPU= 20.872
00:08:53 STEP 20 H= 20:00 +CPU= 21.882
00:09:26 STEP 21 H= 21:00 +CPU= 32.834
00:09:48 STEP 22 H= 22:00 +CPU= 21.208
00:10:09 STEP 23 H= 23:00 +CPU= 21.416
00:10:49 STEP 24 H= 24:00 +CPU= 38.446
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000024
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000000
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000012
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000012
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715_0.zip
Uploading the intermediate file: upload_file_0.zip
00:11:10 STEP 25 H= 25:00 +CPU= 21.041
00:11:32 STEP 26 H= 26:00 +CPU= 21.743
00:12:06 STEP 27 H= 27:00 +CPU= 32.804
00:12:28 STEP 28 H= 28:00 +CPU= 21.403
00:12:49 STEP 29 H= 29:00 +CPU= 20.964
00:13:23 STEP 30 H= 30:00 +CPU= 33.463
00:13:44 STEP 31 H= 31:00 +CPU= 20.643
00:14:06 STEP 32 H= 32:00 +CPU= 21.735
00:14:40 STEP 33 H= 33:00 +CPU= 33.309
00:15:02 STEP 34 H= 34:00 +CPU= 21.267
00:15:23 STEP 35 H= 35:00 +CPU= 21.023
00:16:00 STEP 36 H= 36:00 +CPU= 35.837
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000036
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000036
00:16:22 STEP 37 H= 37:00 +CPU= 21.316
00:16:44 STEP 38 H= 38:00 +CPU= 21.491
00:17:17 STEP 39 H= 39:00 +CPU= 32.959
00:17:39 STEP 40 H= 40:00 +CPU= 21.383
00:18:00 STEP 41 H= 41:00 +CPU= 20.703
00:18:34 STEP 42 H= 42:00 +CPU= 33.522
00:18:56 STEP 43 H= 43:00 +CPU= 20.850
00:19:18 STEP 44 H= 44:00 +CPU= 21.477
00:19:51 STEP 45 H= 45:00 +CPU= 32.905
00:20:13 STEP 46 H= 46:00 +CPU= 21.273
00:20:35 STEP 47 H= 47:00 +CPU= 21.467
00:21:14 STEP 48 H= 48:00 +CPU= 38.686
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000048
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000024
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000036
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000036
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715_1.zip
Uploading the intermediate file: upload_file_1.zip
00:21:35 STEP 49 H= 49:00 +CPU= 20.896
00:21:57 STEP 50 H= 50:00 +CPU= 21.049
00:22:30 STEP 51 H= 51:00 +CPU= 32.516
00:22:52 STEP 52 H= 52:00 +CPU= 21.277
00:23:13 STEP 53 H= 53:00 +CPU= 20.793
00:23:47 STEP 54 H= 54:00 +CPU= 33.725
00:24:08 STEP 55 H= 55:00 +CPU= 20.857
00:24:29 STEP 56 H= 56:00 +CPU= 20.971
00:25:02 STEP 57 H= 57:00 +CPU= 32.395
00:25:25 STEP 58 H= 58:00 +CPU= 22.031
00:25:45 STEP 59 H= 59:00 +CPU= 20.274
00:26:22 STEP 60 H= 60:00 +CPU= 36.508
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000060
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000060
00:26:44 STEP 61 H= 61:00 +CPU= 20.741
00:27:06 STEP 62 H= 62:00 +CPU= 21.753
00:27:39 STEP 63 H= 63:00 +CPU= 32.907
00:28:01 STEP 64 H= 64:00 +CPU= 20.868
00:28:22 STEP 65 H= 65:00 +CPU= 20.721
00:28:56 STEP 66 H= 66:00 +CPU= 33.484
00:29:16 STEP 67 H= 67:00 +CPU= 20.643
00:29:37 STEP 68 H= 68:00 +CPU= 20.693
00:30:11 STEP 69 H= 69:00 +CPU= 32.736
00:30:32 STEP 70 H= 70:00 +CPU= 21.224
00:30:54 STEP 71 H= 71:00 +CPU= 21.681
00:31:34 STEP 72 H= 72:00 +CPU= 38.825
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000072
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000048
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000060
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000060
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715_2.zip
Uploading the intermediate file: upload_file_2.zip
00:31:55 STEP 73 H= 73:00 +CPU= 20.934
00:32:16 STEP 74 H= 74:00 +CPU= 21.200
00:32:49 STEP 75 H= 75:00 +CPU= 32.248
00:33:11 STEP 76 H= 76:00 +CPU= 21.452
00:33:32 STEP 77 H= 77:00 +CPU= 20.548
00:34:05 STEP 78 H= 78:00 +CPU= 32.995
00:34:26 STEP 79 H= 79:00 +CPU= 20.777
00:34:48 STEP 80 H= 80:00 +CPU= 21.876
00:35:22 STEP 81 H= 81:00 +CPU= 32.862
00:35:43 STEP 82 H= 82:00 +CPU= 21.205
00:36:04 STEP 83 H= 83:00 +CPU= 20.729
00:36:41 STEP 84 H= 84:00 +CPU= 36.469
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000084
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000084
00:37:02 STEP 85 H= 85:00 +CPU= 20.500
00:37:24 STEP 86 H= 86:00 +CPU= 21.549
00:37:57 STEP 87 H= 87:00 +CPU= 32.753
00:38:19 STEP 88 H= 88:00 +CPU= 21.190
00:38:39 STEP 89 H= 89:00 +CPU= 20.413
00:39:15 STEP 90 H= 90:00 +CPU= 34.472
00:39:35 STEP 91 H= 91:00 +CPU= 20.423
00:39:57 STEP 92 H= 92:00 +CPU= 21.587
00:40:30 STEP 93 H= 93:00 +CPU= 32.613
00:40:52 STEP 94 H= 94:00 +CPU= 21.483
00:41:14 STEP 95 H= 95:00 +CPU= 21.236
00:41:54 STEP 96 H= 96:00 +CPU= 39.264
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000096
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000084
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000084
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715_3.zip
Uploading the intermediate file: upload_file_3.zip
00:42:14 STEP 97 H= 97:00 +CPU= 20.606
00:42:36 STEP 98 H= 98:00 +CPU= 21.205
00:43:09 STEP 99 H= 99:00 +CPU= 32.899
00:43:31 STEP 100 H= 100:00 +CPU= 21.375
00:43:52 STEP 101 H= 101:00 +CPU= 20.493
00:44:25 STEP 102 H= 102:00 +CPU= 33.234
00:44:47 STEP 103 H= 103:00 +CPU= 20.831
00:45:08 STEP 104 H= 104:00 +CPU= 21.450
00:45:41 STEP 105 H= 105:00 +CPU= 32.597
00:46:03 STEP 106 H= 106:00 +CPU= 21.230
00:46:24 STEP 107 H= 107:00 +CPU= 20.715
00:47:00 STEP 108 H= 108:00 +CPU= 36.015
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000108
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000108
00:47:22 STEP 109 H= 109:00 +CPU= 20.817
00:47:43 STEP 110 H= 110:00 +CPU= 21.408
00:48:17 STEP 111 H= 111:00 +CPU= 32.777
00:48:38 STEP 112 H= 112:00 +CPU= 21.337
00:48:59 STEP 113 H= 113:00 +CPU= 20.623
00:49:33 STEP 114 H= 114:00 +CPU= 33.294
00:49:54 STEP 115 H= 115:00 +CPU= 20.682
00:50:16 STEP 116 H= 116:00 +CPU= 21.458
00:50:48 STEP 117 H= 117:00 +CPU= 32.128
00:51:10 STEP 118 H= 118:00 +CPU= 21.472
00:51:32 STEP 119 H= 119:00 +CPU= 21.051
00:52:11 STEP 120 H= 120:00 +CPU= 38.822
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMGGhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMSHhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300/ICMUAhq0f+000108
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715_4.zip
Uploading the intermediate file: upload_file_4.zip
00:52:32 STEP 121 H= 121:00 +CPU= 20.955
00:52:54 STEP 122 H= 122:00 +CPU= 21.436
00:53:27 STEP 123 H= 123:00 +CPU= 32.324
00:53:48 STEP 124 H= 124:00 +CPU= 20.736
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0656
(argv4) batchid: 952
(argv5) wuid: 12169300
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169300
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2686d596a53b64a6b0d7aa9cf38688fa to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0656_1983050100_123_952_12169300.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0656_1983050100_123_952_12169300.zip
ic_ancil_file: ic_ancil_12169300
ifsdata_file: ifsdata_12169300
climate_data_file: clim_data_12169300
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_051dea1693c95b80c1b9121ce69d77fd to: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12169300.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4046bbe13aea1bbc3b8858012b09cd85 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12169300.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12169300.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12169300.xml
-- Model is restarting --
Adjusting last_iter, 125, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12169300.xml
last_cpu_time: 3202
upload_file_number: 5
last_iter: 119
last_upload: 432000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0656_1983050100_123_952_12169300_2_r1848188715
The child process has been launched with process id: 1624334
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Sibe-Ubuntu:1:1:1624334:1624334] [20230112:010010:1673481610:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
01:00:49 STEP 120 H= 120:00 +CPU= 34.655
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
