climateprediction.net home page
Task 22267865

Task 22267865

Name oifs_43r3_ps_0101_1997050100_123_966_12182745_0
Workunit 12182745
Created 22 Dec 2022, 12:42:57 UTC
Sent 3 Jan 2023, 18:06:34 UTC
Report deadline 2 Feb 2023, 18:06:34 UTC
Received 8 Jan 2023, 4:23:03 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1534746
Run time 4 hours 7 min 14 sec
CPU time 7 min 56 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 5.61 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 3,364.04 MB
Peak swap size 3,988.23 MB
Peak disk usage 438.40 MB
Stderr 
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)</message>
<stderr_txt>
9b48b9a2e to: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Checking for progress XML file: /var/lib/boinc/slots/36/progress_file_12182745.xml
Creating progress file: /var/lib/boinc/slots/36/progress_file_12182745.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0101_1997050100_123_966_12182745_0_r493789893
The child process has been launched with process id: 8209
Executing the command: /var/lib/boinc/slots/36/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8209:8209] [20230107:095434:1673060074:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/36/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc/slots/36/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc/slots/36/ICMUAhq0f+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:31:34 STEP    0 H=   0:00 +CPU=157.757
Moving to projects directory: /var/lib/boinc/slots/36/ICMGGhq0f+000000
Moving to projects directory: /var/lib/boinc/slots/36/ICMSHhq0f+000000
Moving to projects directory: /var/lib/boinc/slots/36/ICMUAhq0f+000000
  13:31:55 STEP    1 H=   1:00 +CPU= 17.464
  13:31:57 STEP    0 H=   0:00 +CPU=177.463
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:56:19 STEP    2 H=   2:00 +CPU= 21.006
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cbbc6f1c69336c2d52186ce82c018303 to: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
ic_ancil_file: ic_ancil_12182745
ifsdata_file: ifsdata_12182745
climate_data_file: clim_data_12182745
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a2bcbaa7adb2422ec8b59d850d21b5e to: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
  15:29:12 STEP    3 H=   3:00 +CPU= 40.254
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4194509c5219dbed1a406992d622d01 to: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
  15:29:32 STEP    4 H=   4:00 +CPU= 19.264
Checking for progress XML file: /var/lib/boinc/slots/36/progress_file_12182745.xml
Opened progress file ok : /var/lib/boinc/slots/36/progress_file_12182745.xml
-- Model is restarting --
Adjusting last_iter, 3, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/36/progress_file_12182745.xml
last_cpu_time: 204
upload_file_number: 0
last_iter: -1
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0101_1997050100_123_966_12182745_0_r493789893
The child process has been launched with process id: 8727
Executing the command: /var/lib/boinc/slots/36/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:8727:8727] [20230107:152934:1673080174:0.028] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:31:49 STEP    5 H=   5:00 +CPU= 29.346
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cbbc6f1c69336c2d52186ce82c018303 to: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
ic_ancil_file: ic_ancil_12182745
ifsdata_file: ifsdata_12182745
climate_data_file: clim_data_12182745
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a2bcbaa7adb2422ec8b59d850d21b5e to: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4194509c5219dbed1a406992d622d01 to: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Checking for progress XML file: /var/lib/boinc/slots/36/progress_file_12182745.xml
Opened progress file ok : /var/lib/boinc/slots/36/progress_file_12182745.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/36/progress_file_12182745.xml
last_cpu_time: 301
upload_file_number: 0
last_iter: -985
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0101_1997050100_123_966_12182745_0_r493789893
The child process has been launched with process id: 9135
Executing the command: /var/lib/boinc/slots/36/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9135:9135] [20230107:193901:1673095141:0.012] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
  19:39:03 STEP    6 H=   6:00 +CPU=225.805
Resuming the child process
  19:39:18 STEP    0 H=   0:00 +CPU=238.758
  19:39:21 STEP    7 H=   7:00 +CPU= 17.608
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
  22:47:44 STEP    1 H=   1:00 +CPU=202.050
  22:47:45 STEP    8 H=   8:00 +CPU=172.949
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cbbc6f1c69336c2d52186ce82c018303 to: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
ic_ancil_file: ic_ancil_12182745
ifsdata_file: ifsdata_12182745
climate_data_file: clim_data_12182745
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a2bcbaa7adb2422ec8b59d850d21b5e to: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4194509c5219dbed1a406992d622d01 to: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
  00:31:53 STEP    2 H=   2:00 +CPU=210.370
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Checking for progress XML file: /var/lib/boinc/slots/36/progress_file_12182745.xml
Opened progress file ok : /var/lib/boinc/slots/36/progress_file_12182745.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/36/progress_file_12182745.xml
last_cpu_time: 373
upload_file_number: 0
last_iter: -985
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0101_1997050100_123_966_12182745_0_r493789893
The child process has been launched with process id: 9777
Executing the command: /var/lib/boinc/slots/36/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:9777:9777] [20230108:003154:1673112714:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
  00:32:02 STEP    0 H=   0:00 +CPU=201.486
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:32:36 STEP    9 H=   9:00 +CPU=145.063
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
  00:32:51 STEP    3 H=   3:00 +CPU= 35.624
  00:32:54 STEP    1 H=   1:00 +CPU= 28.271
Suspend request received from the BOINC client, suspending the child process
  00:33:52 STEP   10 H=  10:00 +CPU= 35.290
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_cbbc6f1c69336c2d52186ce82c018303 to: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/36/oifs_43r3_ps_0101_1997050100_123_966_12182745.zip
ic_ancil_file: ic_ancil_12182745
ifsdata_file: ifsdata_12182745
climate_data_file: clim_data_12182745
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a2bcbaa7adb2422ec8b59d850d21b5e to: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/36/ic_ancil_12182745.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c4194509c5219dbed1a406992d622d01 to: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/36/ifsdata/ifsdata_12182745.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/36/159l_2/clim_data_12182745.zip
  08:51:28 STEP    2 H=   2:00 +CPU=105.978
Checking for progress XML file: /var/lib/boinc/slots/36/progress_file_12182745.xml
Opened progress file ok : /var/lib/boinc/slots/36/progress_file_12182745.xml
-- Model is restarting --
Adjusting last_iter, -985, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/36/progress_file_12182745.xml
last_cpu_time: 397
upload_file_number: 0
last_iter: -985
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0101_1997050100_123_966_12182745_0_r493789893
The child process has been launched with process id: 10440
Executing the command: /var/lib/boinc/slots/36/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:10440:10440] [20230108:085130:1673142690:0.052] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:51:30 STEP   11 H=  11:00 +CPU= 99.975
  08:51:37 STEP    0 H=   0:00 +CPU=161.200
Moving to projects directory: /var/lib/boinc/slots/36/ICMGGhq0f+000012
Moving to projects directory: /var/lib/boinc/slots/36/ICMSHhq0f+000012
Moving to projects directory: /var/lib/boinc/slots/36/ICMUAhq0f+000012
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:56:46 STEP    3 H=   3:00 +CPU= 48.030
  08:57:15 STEP    1 H=   1:00 +CPU= 50.199
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Invalid characters in stoi string: STEPO
Unable to update iter, resetting to last_iter.
ifs_line = 
Suspend request received from the BOINC client, suspending the child process
  09:18:30 STEP    4 H=   4:00 +CPU= 53.454
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
SIGSEGV: segmentation violation
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0101
(argv4) batchid: 966
(argv5) wuid: 12182745
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/36
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12182745
Stack trace (8 frames):
[0x453b63]
[0x53c3f0]
[0x452c07]
[0x41cf1c]
[0x41f305]
[0x41f85e]
..mkdir for temp folder for results failed
[0x538cf9]
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/36/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
[0x5fb433]

Exiting...

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net