<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.021] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.021] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.021] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.021] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.021] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.021] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:9842:9842] [20221228:102411:1672194251:0.022] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0485
(argv4) batchid: 973
(argv5) wuid: 12190129
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5ff2dadc629e67cf15a1a2fb6428f3e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
ic_ancil_file: ic_ancil_12190129
ifsdata_file: ifsdata_12190129
climate_data_file: clim_data_12190129
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e3cf9cdc58a606dc80a01abb9e6a074b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12190129.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
last_cpu_time: 711
upload_file_number: 1
last_iter: -985
last_upload: 1378800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955
The child process has been launched with process id: 10158
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:10158:10158] [20221228:103454:1672194894:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0485
(argv4) batchid: 973
(argv5) wuid: 12190129
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5ff2dadc629e67cf15a1a2fb6428f3e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
ic_ancil_file: ic_ancil_12190129
ifsdata_file: ifsdata_12190129
climate_data_file: clim_data_12190129
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e3cf9cdc58a606dc80a01abb9e6a074b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12190129.xml
-- Model is restarting --
Adjusting last_iter, 384, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
last_cpu_time: 721
upload_file_number: 1
last_iter: 383
last_upload: 1378800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955
The child process has been launched with process id: 24692
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24692:24692] [20221228:173528:1672220128:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0485
(argv4) batchid: 973
(argv5) wuid: 12190129
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5ff2dadc629e67cf15a1a2fb6428f3e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
ic_ancil_file: ic_ancil_12190129
ifsdata_file: ifsdata_12190129
climate_data_file: clim_data_12190129
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e3cf9cdc58a606dc80a01abb9e6a074b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12190129.xml
-- Model is restarting --
Adjusting last_iter, 384, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
last_cpu_time: 750
upload_file_number: 1
last_iter: 383
last_upload: 1378800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955
The child process has been launched with process id: 24993
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:24993:24993] [20221228:174552:1672220752:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:48:02 STEP 384 H= 384:00 +CPU= 51.894
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:52:03 STEP 385 H= 385:00 +CPU= 31.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:54:53 STEP 386 H= 386:00 +CPU= 29.361
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:58:28 STEP 387 H= 387:00 +CPU= 45.372
17:59:10 STEP 388 H= 388:00 +CPU= 27.009
18:01:57 STEP 389 H= 389:00 +CPU= 29.410
18:06:19 STEP 390 H= 390:00 +CPU= 47.356
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:09:52 STEP 391 H= 391:00 +CPU= 31.503
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:12:32 STEP 392 H= 392:00 +CPU= 30.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:15:50 STEP 393 H= 393:00 +CPU= 43.711
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:18:40 STEP 394 H= 394:00 +CPU= 30.405
18:19:30 STEP 395 H= 395:00 +CPU= 27.883
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:23:07 STEP 396 H= 396:00 +CPU= 50.195
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:26:20 STEP 397 H= 397:00 +CPU= 32.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:31:08 STEP 398 H= 398:00 +CPU= 34.454
18:34:56 STEP 399 H= 399:00 +CPU= 44.817
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:43:22 STEP 400 H= 400:00 +CPU= 36.348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:54:38 STEP 401 H= 401:00 +CPU= 38.943
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:13:36 STEP 402 H= 402:00 +CPU= 62.472
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:27:15 STEP 403 H= 403:00 +CPU= 37.146
19:38:46 STEP 404 H= 404:00 +CPU= 41.713
19:57:06 STEP 405 H= 405:00 +CPU= 58.192
20:11:27 STEP 406 H= 406:00 +CPU= 42.022
20:17:07 STEP 407 H= 407:00 +CPU= 26.759
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955_1.zip
Uploading the intermediate file: upload_file_1.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:18:54 STEP 408 H= 408:00 +CPU= 43.033
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000408
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:19:56 STEP 409 H= 409:00 +CPU= 26.551
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:20:47 STEP 410 H= 410:00 +CPU= 26.577
20:21:29 STEP 411 H= 411:00 +CPU= 34.206
20:22:35 STEP 412 H= 412:00 +CPU= 26.219
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:23:23 STEP 413 H= 413:00 +CPU= 24.511
20:24:34 STEP 414 H= 414:00 +CPU= 42.354
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:26:03 STEP 415 H= 415:00 +CPU= 27.925
20:27:00 STEP 416 H= 416:00 +CPU= 27.435
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:28:27 STEP 417 H= 417:00 +CPU= 39.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:29:37 STEP 418 H= 418:00 +CPU= 27.326
20:30:29 STEP 419 H= 419:00 +CPU= 24.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:31:58 STEP 420 H= 420:00 +CPU= 41.492
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000420
20:32:23 STEP 421 H= 421:00 +CPU= 21.598
20:33:31 STEP 422 H= 422:00 +CPU= 26.236
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:34:56 STEP 423 H= 423:00 +CPU= 37.461
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:35:59 STEP 424 H= 424:00 +CPU= 24.242
20:36:33 STEP 425 H= 425:00 +CPU= 25.582
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:37:51 STEP 426 H= 426:00 +CPU= 41.192
20:38:56 STEP 427 H= 427:00 +CPU= 24.900
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:40:15 STEP 428 H= 428:00 +CPU= 28.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:41:44 STEP 429 H= 429:00 +CPU= 36.975
20:42:40 STEP 430 H= 430:00 +CPU= 27.011
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:43:22 STEP 431 H= 431:00 +CPU= 23.355
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955_2.zip
Uploading the intermediate file: upload_file_2.zip
20:45:05 STEP 432 H= 432:00 +CPU= 44.072
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000432
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:46:02 STEP 433 H= 433:00 +CPU= 26.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:47:07 STEP 434 H= 434:00 +CPU= 26.998
20:48:50 STEP 435 H= 435:00 +CPU= 37.732
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:50:01 STEP 436 H= 436:00 +CPU= 27.056
20:50:32 STEP 437 H= 437:00 +CPU= 26.223
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:51:56 STEP 438 H= 438:00 +CPU= 38.504
20:52:51 STEP 439 H= 439:00 +CPU= 24.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:53:55 STEP 440 H= 440:00 +CPU= 27.021
20:55:46 STEP 441 H= 441:00 +CPU= 39.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:56:50 STEP 442 H= 442:00 +CPU= 26.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:57:56 STEP 443 H= 443:00 +CPU= 26.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:59:27 STEP 444 H= 444:00 +CPU= 44.849
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000444
21:00:46 STEP 445 H= 445:00 +CPU= 27.237
21:01:47 STEP 446 H= 446:00 +CPU= 25.990
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:03:16 STEP 447 H= 447:00 +CPU= 41.352
21:04:09 STEP 448 H= 448:00 +CPU= 26.129
21:05:11 STEP 449 H= 449:00 +CPU= 26.453
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:06:42 STEP 450 H= 450:00 +CPU= 40.221
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:07:49 STEP 451 H= 451:00 +CPU= 25.090
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:08:56 STEP 452 H= 452:00 +CPU= 25.277
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:10:34 STEP 453 H= 453:00 +CPU= 40.250
21:11:34 STEP 454 H= 454:00 +CPU= 27.532
21:12:01 STEP 455 H= 455:00 +CPU= 21.333
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955_3.zip
Uploading the intermediate file: upload_file_3.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:13:51 STEP 456 H= 456:00 +CPU= 45.961
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000456
21:15:13 STEP 457 H= 457:00 +CPU= 25.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:16:20 STEP 458 H= 458:00 +CPU= 24.828
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0485
(argv4) batchid: 973
(argv5) wuid: 12190129
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5ff2dadc629e67cf15a1a2fb6428f3e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
ic_ancil_file: ic_ancil_12190129
ifsdata_file: ifsdata_12190129
climate_data_file: clim_data_12190129
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e3cf9cdc58a606dc80a01abb9e6a074b to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12190129.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12190129.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12190129.xml
-- Model is restarting --
Adjusting last_iter, 459, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12190129.xml
last_cpu_time: 3265
upload_file_number: 4
last_iter: 455
last_upload: 1638000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955
The child process has been launched with process id: 20564
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:X450J:1:1:20564:20564] [20221229:173901:1672306741:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
17:41:42 STEP 456 H= 456:00 +CPU= 46.209
17:45:11 STEP 457 H= 457:00 +CPU= 28.084
17:47:23 STEP 458 H= 458:00 +CPU= 25.363
17:48:18 STEP 459 H= 459:00 +CPU= 35.886
17:50:26 STEP 460 H= 460:00 +CPU= 24.500
17:50:51 STEP 461 H= 461:00 +CPU= 21.046
17:51:33 STEP 462 H= 462:00 +CPU= 33.986
17:53:05 STEP 463 H= 463:00 +CPU= 23.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:53:43 STEP 464 H= 464:00 +CPU= 22.023
17:54:21 STEP 465 H= 465:00 +CPU= 33.189
17:56:06 STEP 466 H= 466:00 +CPU= 25.276
17:56:31 STEP 467 H= 467:00 +CPU= 20.364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:58:12 STEP 468 H= 468:00 +CPU= 41.133
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000468
17:59:15 STEP 469 H= 469:00 +CPU= 24.362
18:00:47 STEP 470 H= 470:00 +CPU= 26.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:02:51 STEP 471 H= 471:00 +CPU= 38.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:04:34 STEP 472 H= 472:00 +CPU= 26.258
18:05:35 STEP 473 H= 473:00 +CPU= 25.963
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:07:40 STEP 474 H= 474:00 +CPU= 40.102
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:10:03 STEP 475 H= 475:00 +CPU= 25.926
18:12:36 STEP 476 H= 476:00 +CPU= 28.297
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:15:02 STEP 477 H= 477:00 +CPU= 40.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:16:53 STEP 478 H= 478:00 +CPU= 27.493
18:18:16 STEP 479 H= 479:00 +CPU= 24.985
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0485_2004050100_123_973_12190129_0_r1112893955_4.zip
Uploading the intermediate file: upload_file_4.zip
18:22:21 STEP 480 H= 480:00 +CPU= 47.681
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000480
18:24:12 STEP 481 H= 481:00 +CPU= 26.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:36:58 STEP 482 H= 482:00 +CPU= 32.619
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:00:44 STEP 483 H= 483:00 +CPU= 59.524
19:15:50 STEP 484 H= 484:00 +CPU= 35.990
19:37:38 STEP 485 H= 485:00 +CPU= 43.015
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0485
(argv4) batchid: 973
(argv5) wuid: 12190129
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5ff2dadc629e67cf15a1a2fb6428f3e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
ic_ancil_file: ic_ancil_12190129
ifsdata_file: ifsdata_12190129
climate_data_file: clim_data_12190129
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0a4111b8bc796e1a52d2b880fd02ff1 to: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
double free or corruption (top)
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0485
(argv4) batchid: 973
(argv5) wuid: 12190129
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190129
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b5ff2dadc629e67cf15a1a2fb6428f3e to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0485_2004050100_123_973_12190129.zip
ic_ancil_file: ic_ancil_12190129
ifsdata_file: ifsdata_12190129
climate_data_file: clim_data_12190129
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ÀÿÉA‘—é¡`§Ø7 to: /var/lib/boinc-client/slots/5/ic_ancil_12190129.zip
..Copying the IC ancils to the working directory failed
</stderr_txt>
]]>
