climateprediction.net home page
Task 22276144

Task 22276144

Name oifs_43r3_ps_0555_2002050100_123_971_12188199_1
Workunit 12188199
Created 23 Dec 2022, 9:04:47 UTC
Sent 10 Jan 2023, 20:47:11 UTC
Report deadline 9 Feb 2023, 20:47:11 UTC
Received 13 Jan 2023, 16:29:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1537813
Run time 1 days 23 hours 48 min 55 sec
CPU time 38 min 11 sec
Validate state Invalid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,405.80 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,853.21 MB
Stderr 
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<stderr_txt>
ef>
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96768:96768] [20230113:080003:1673618403:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0555
(argv4) batchid: 971
(argv5) wuid: 12188199
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23abe25be009a92614143af357154685 to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
ic_ancil_file: ic_ancil_12188199
ifsdata_file: ifsdata_12188199
climate_data_file: clim_data_12188199
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9bf2a5771d095e15ebd8aaf8d6272ac to: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4372651a3407c02b445a71b0adc677e8 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12188199.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
last_cpu_time: 170200
upload_file_number: 122
last_iter: -985
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087
The child process has been launched with process id: 96851
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:96851:96851] [20230113:080726:1673618846:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0555
(argv4) batchid: 971
(argv5) wuid: 12188199
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23abe25be009a92614143af357154685 to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
ic_ancil_file: ic_ancil_12188199
ifsdata_file: ifsdata_12188199
climate_data_file: clim_data_12188199
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9bf2a5771d095e15ebd8aaf8d6272ac to: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4372651a3407c02b445a71b0adc677e8 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12188199.xml
-- Model is restarting --
Adjusting last_iter, 2928, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
last_cpu_time: 170229
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087
The child process has been launched with process id: 97092
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97092:97092] [20230113:082218:1673619738:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0555
(argv4) batchid: 971
(argv5) wuid: 12188199
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23abe25be009a92614143af357154685 to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
ic_ancil_file: ic_ancil_12188199
ifsdata_file: ifsdata_12188199
climate_data_file: clim_data_12188199
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9bf2a5771d095e15ebd8aaf8d6272ac to: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4372651a3407c02b445a71b0adc677e8 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087
The child process has been launched with process id: 97129
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97129:97129] [20230113:082426:1673619866:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0555
(argv4) batchid: 971
(argv5) wuid: 12188199
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23abe25be009a92614143af357154685 to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
ic_ancil_file: ic_ancil_12188199
ifsdata_file: ifsdata_12188199
climate_data_file: clim_data_12188199
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9bf2a5771d095e15ebd8aaf8d6272ac to: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4372651a3407c02b445a71b0adc677e8 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12188199.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
last_cpu_time: 8
upload_file_number: 0
last_iter: -985
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087
The child process has been launched with process id: 97155
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97155:97155] [20230113:082546:1673619946:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0555
(argv4) batchid: 971
(argv5) wuid: 12188199
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23abe25be009a92614143af357154685 to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
ic_ancil_file: ic_ancil_12188199
ifsdata_file: ifsdata_12188199
climate_data_file: clim_data_12188199
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9bf2a5771d095e15ebd8aaf8d6272ac to: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4372651a3407c02b445a71b0adc677e8 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12188199.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
last_cpu_time: 17
upload_file_number: 0
last_iter: -985
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087
The child process has been launched with process id: 97181
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97181:97181] [20230113:082706:1673620026:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0555
(argv4) batchid: 971
(argv5) wuid: 12188199
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_23abe25be009a92614143af357154685 to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0555_2002050100_123_971_12188199.zip
ic_ancil_file: ic_ancil_12188199
ifsdata_file: ifsdata_12188199
climate_data_file: clim_data_12188199
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b9bf2a5771d095e15ebd8aaf8d6272ac to: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12188199.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4372651a3407c02b445a71b0adc677e8 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12188199.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12188199.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12188199.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12188199.xml
last_cpu_time: 27
upload_file_number: 0
last_iter: -985
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087
The child process has been launched with process id: 97488
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc4:1:1:97488:97488] [20230113:084535:1673621135:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:48:07 STEP 2928 H=2928:00 +CPU=139.368
  08:49:17 STEP 2929 H=2929:00 +CPU= 69.576
  08:50:30 STEP 2930 H=2930:00 +CPU= 72.544
  08:52:07 STEP 2931 H=2931:00 +CPU= 96.645
  08:53:12 STEP 2932 H=2932:00 +CPU= 64.843
  08:54:14 STEP 2933 H=2933:00 +CPU= 62.047
  08:55:55 STEP 2934 H=2934:00 +CPU=100.849
  08:56:59 STEP 2935 H=2935:00 +CPU= 64.551
  08:58:02 STEP 2936 H=2936:00 +CPU= 62.977
  08:59:54 STEP 2937 H=2937:00 +CPU=111.228
  09:01:05 STEP 2938 H=2938:00 +CPU= 71.107
  09:02:18 STEP 2939 H=2939:00 +CPU= 72.295
  09:04:28 STEP 2940 H=2940:00 +CPU=130.199
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002940
  09:05:41 STEP 2941 H=2941:00 +CPU= 72.449
  09:06:52 STEP 2942 H=2942:00 +CPU= 71.296
  09:08:46 STEP 2943 H=2943:00 +CPU=114.329
  09:10:12 STEP 2944 H=2944:00 +CPU= 85.239
  09:11:46 STEP 2945 H=2945:00 +CPU= 93.873
  09:14:02 STEP 2946 H=2946:00 +CPU=135.208
  09:15:24 STEP 2947 H=2947:00 +CPU= 81.923
  09:16:45 STEP 2948 H=2948:00 +CPU= 81.918
  09:18:49 STEP 2949 H=2949:00 +CPU=123.604
  09:19:58 STEP 2950 H=2950:00 +CPU= 69.269
  09:21:11 STEP 2951 H=2951:00 +CPU= 72.094
  09:23:25 STEP 2952 H=2952:00 +CPU=133.999
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188199/ICMSHhq0f+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087_1.zip
Uploading the final file: upload_file_1.zip
Uploading trickle at timestep: 10623600
09:26:51 (97480): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_ps_0555_2002050100_123_971_12188199_1_r589405087_122.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:18:08 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 10,623,600 2,291 0.0002
09 May 2023 16:17:58 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 10,368,000 165,815 0.0160
09 May 2023 16:17:45 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 9,504,000 152,834 0.0161
09 May 2023 16:17:36 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 8,640,000 139,796 0.0162
09 May 2023 16:17:23 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 7,776,000 126,462 0.0163
09 May 2023 16:17:11 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 6,912,000 113,301 0.0164
09 May 2023 16:16:38 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 6,048,000 99,997 0.0165
09 May 2023 16:16:26 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 5,184,000 86,044 0.0166
09 May 2023 16:16:12 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 4,320,000 72,178 0.0167
09 May 2023 16:15:59 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 3,456,000 58,224 0.0168
09 May 2023 16:15:47 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 2,592,000 43,375 0.0167
09 May 2023 16:15:37 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 1,728,000 28,679 0.0166
09 May 2023 16:15:13 1537813 22276144 oifs_43r3_ps_0555_2002050100_123_971_12188199_1 864,000 14,647 0.0170


©2024 climateprediction.net