climateprediction.net home page
Task 22293431

Task 22293431

Name oifs_43r3_ps_0271_2005050100_123_974_12190915_1
Workunit 12190915
Created 25 Dec 2022, 11:03:10 UTC
Sent 6 Jan 2023, 13:03:23 UTC
Report deadline 5 Feb 2023, 13:03:23 UTC
Received 8 Jan 2023, 0:28:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1491267
Run time 23 hours 31 min
CPU time 20 hours 52 min 34 sec
Validate state Invalid
Credit 7,620.48
Device peak FLOPS 4.56 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,436.95 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,168.27 MB
Stderr 
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
53:51 STEP 1611 H=1611:00 +CPU= 33.856
  20:54:17 STEP 1612 H=1612:00 +CPU= 25.838
  20:57:51 STEP 1613 H=1613:00 +CPU= 25.508
  20:58:33 STEP 1614 H=1614:00 +CPU= 31.756
  20:58:54 STEP 1615 H=1615:00 +CPU= 20.228
  20:59:16 STEP 1616 H=1616:00 +CPU= 21.866
  20:59:58 STEP 1617 H=1617:00 +CPU= 41.442
  21:04:27 STEP 1618 H=1618:00 +CPU= 30.470
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0271
(argv4) batchid: 974
(argv5) wuid: 12190915
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6841f1b36ed6c39b8a368254b1c46880 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
ic_ancil_file: ic_ancil_12190915
ifsdata_file: ifsdata_12190915
climate_data_file: clim_data_12190915
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ca417744e42b7dae45aab4d8ad39f288 to: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_dc203379185d3a89bdc7c8ac732e9cb0 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12190915.xml
-- Model is restarting --
Adjusting last_iter, 1618, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
last_cpu_time: 64119
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461
The child process has been launched with process id: 343980
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:343980:343980] [20230107:210851:1673125731:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:10:47 STEP 1608 H=1608:00 +CPU= 79.629
  21:12:33 STEP 1609 H=1609:00 +CPU= 43.306
  21:13:22 STEP 1610 H=1610:00 +CPU= 48.628
  21:14:29 STEP 1611 H=1611:00 +CPU= 63.690
  21:15:17 STEP 1612 H=1612:00 +CPU= 46.495
  21:16:06 STEP 1613 H=1613:00 +CPU= 48.184
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0271
(argv4) batchid: 974
(argv5) wuid: 12190915
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6841f1b36ed6c39b8a368254b1c46880 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
ic_ancil_file: ic_ancil_12190915
ifsdata_file: ifsdata_12190915
climate_data_file: clim_data_12190915
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ca417744e42b7dae45aab4d8ad39f288 to: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_dc203379185d3a89bdc7c8ac732e9cb0 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12190915.xml
-- Model is restarting --
Adjusting last_iter, 1614, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
last_cpu_time: 64487
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461
The child process has been launched with process id: 344017
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344017:344017] [20230107:211750:1673126270:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:19:25 STEP 1608 H=1608:00 +CPU= 75.398
  21:20:12 STEP 1609 H=1609:00 +CPU= 45.994
  21:21:04 STEP 1610 H=1610:00 +CPU= 50.850
  21:22:18 STEP 1611 H=1611:00 +CPU= 72.292
  21:23:06 STEP 1612 H=1612:00 +CPU= 47.605
  21:23:51 STEP 1613 H=1613:00 +CPU= 45.063
  21:24:54 STEP 1614 H=1614:00 +CPU= 61.811
  21:25:37 STEP 1615 H=1615:00 +CPU= 41.780
  21:26:20 STEP 1616 H=1616:00 +CPU= 42.652
  21:27:24 STEP 1617 H=1617:00 +CPU= 64.179
  21:28:08 STEP 1618 H=1618:00 +CPU= 42.932
  21:28:49 STEP 1619 H=1619:00 +CPU= 41.073
  21:29:57 STEP 1620 H=1620:00 +CPU= 65.941
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001620
  21:30:42 STEP 1621 H=1621:00 +CPU= 44.297
  21:31:33 STEP 1622 H=1622:00 +CPU= 50.544
  21:32:51 STEP 1623 H=1623:00 +CPU= 76.898
  21:33:43 STEP 1624 H=1624:00 +CPU= 51.752
  21:34:34 STEP 1625 H=1625:00 +CPU= 50.472
  21:35:49 STEP 1626 H=1626:00 +CPU= 73.443
  21:36:36 STEP 1627 H=1627:00 +CPU= 46.550
  21:37:28 STEP 1628 H=1628:00 +CPU= 51.269
  21:38:37 STEP 1629 H=1629:00 +CPU= 68.475
  21:39:25 STEP 1630 H=1630:00 +CPU= 47.584
  21:40:13 STEP 1631 H=1631:00 +CPU= 47.063
  21:41:25 STEP 1632 H=1632:00 +CPU= 69.445
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_67.zip
Uploading the intermediate file: upload_file_67.zip
  21:42:03 STEP 1633 H=1633:00 +CPU= 37.586
  21:42:42 STEP 1634 H=1634:00 +CPU= 38.609
  21:43:43 STEP 1635 H=1635:00 +CPU= 60.136
  21:44:22 STEP 1636 H=1636:00 +CPU= 38.842
  21:44:57 STEP 1637 H=1637:00 +CPU= 34.887
  21:45:50 STEP 1638 H=1638:00 +CPU= 52.853
  21:46:22 STEP 1639 H=1639:00 +CPU= 32.006
  21:46:59 STEP 1640 H=1640:00 +CPU= 36.673
  21:47:49 STEP 1641 H=1641:00 +CPU= 49.965
  21:48:25 STEP 1642 H=1642:00 +CPU= 35.148
  21:48:59 STEP 1643 H=1643:00 +CPU= 33.924
  21:49:54 STEP 1644 H=1644:00 +CPU= 55.449
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001644
  21:50:30 STEP 1645 H=1645:00 +CPU= 35.145
  21:51:05 STEP 1646 H=1646:00 +CPU= 35.143
  21:51:57 STEP 1647 H=1647:00 +CPU= 51.745
  21:52:33 STEP 1648 H=1648:00 +CPU= 36.194
  21:53:06 STEP 1649 H=1649:00 +CPU= 32.634
  21:53:54 STEP 1650 H=1650:00 +CPU= 47.669
  21:54:24 STEP 1651 H=1651:00 +CPU= 29.456
  21:54:53 STEP 1652 H=1652:00 +CPU= 29.491
  21:55:37 STEP 1653 H=1653:00 +CPU= 43.812
  21:56:06 STEP 1654 H=1654:00 +CPU= 28.843
  21:56:36 STEP 1655 H=1655:00 +CPU= 29.507
  21:57:28 STEP 1656 H=1656:00 +CPU= 51.859
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_68.zip
Uploading the intermediate file: upload_file_68.zip
  21:57:57 STEP 1657 H=1657:00 +CPU= 28.608
  21:58:27 STEP 1658 H=1658:00 +CPU= 29.524
  21:59:11 STEP 1659 H=1659:00 +CPU= 43.663
  21:59:40 STEP 1660 H=1660:00 +CPU= 29.085
  22:00:08 STEP 1661 H=1661:00 +CPU= 28.196
  22:00:53 STEP 1662 H=1662:00 +CPU= 44.686
  22:01:21 STEP 1663 H=1663:00 +CPU= 28.194
  22:01:51 STEP 1664 H=1664:00 +CPU= 29.346
  22:02:35 STEP 1665 H=1665:00 +CPU= 43.986
  22:03:05 STEP 1666 H=1666:00 +CPU= 29.489
  22:03:34 STEP 1667 H=1667:00 +CPU= 28.633
  22:04:22 STEP 1668 H=1668:00 +CPU= 48.576
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001668
  22:04:51 STEP 1669 H=1669:00 +CPU= 28.509
  22:05:20 STEP 1670 H=1670:00 +CPU= 28.922
  22:06:04 STEP 1671 H=1671:00 +CPU= 43.266
  22:06:33 STEP 1672 H=1672:00 +CPU= 29.135
  22:07:02 STEP 1673 H=1673:00 +CPU= 28.704
  22:07:46 STEP 1674 H=1674:00 +CPU= 44.483
  22:08:15 STEP 1675 H=1675:00 +CPU= 28.601
  22:08:44 STEP 1676 H=1676:00 +CPU= 29.108
  22:09:28 STEP 1677 H=1677:00 +CPU= 43.463
  22:09:58 STEP 1678 H=1678:00 +CPU= 29.527
  22:10:27 STEP 1679 H=1679:00 +CPU= 29.294
  22:11:18 STEP 1680 H=1680:00 +CPU= 50.952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_69.zip
Uploading the intermediate file: upload_file_69.zip
  22:11:47 STEP 1681 H=1681:00 +CPU= 28.600
  22:12:17 STEP 1682 H=1682:00 +CPU= 29.379
  22:13:01 STEP 1683 H=1683:00 +CPU= 43.975
  22:13:31 STEP 1684 H=1684:00 +CPU= 29.903
  22:13:59 STEP 1685 H=1685:00 +CPU= 28.483
  22:14:45 STEP 1686 H=1686:00 +CPU= 45.355
  22:15:14 STEP 1687 H=1687:00 +CPU= 28.568
  22:15:43 STEP 1688 H=1688:00 +CPU= 29.313
  22:16:27 STEP 1689 H=1689:00 +CPU= 43.852
  22:16:56 STEP 1690 H=1690:00 +CPU= 28.957
  22:17:25 STEP 1691 H=1691:00 +CPU= 28.226
  22:18:13 STEP 1692 H=1692:00 +CPU= 48.027
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001692
  22:18:42 STEP 1693 H=1693:00 +CPU= 28.811
  22:19:11 STEP 1694 H=1694:00 +CPU= 29.534
  22:19:55 STEP 1695 H=1695:00 +CPU= 43.779
  22:20:25 STEP 1696 H=1696:00 +CPU= 29.290
  22:20:53 STEP 1697 H=1697:00 +CPU= 28.370
  22:21:38 STEP 1698 H=1698:00 +CPU= 44.044
  22:22:07 STEP 1699 H=1699:00 +CPU= 28.830
  22:22:36 STEP 1700 H=1700:00 +CPU= 29.185
  22:23:19 STEP 1701 H=1701:00 +CPU= 43.410
  22:23:49 STEP 1702 H=1702:00 +CPU= 29.181
  22:24:18 STEP 1703 H=1703:00 +CPU= 29.328
  22:25:10 STEP 1704 H=1704:00 +CPU= 51.269
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_70.zip
Uploading the intermediate file: upload_file_70.zip
  22:25:38 STEP 1705 H=1705:00 +CPU= 28.616
  22:26:08 STEP 1706 H=1706:00 +CPU= 29.448
  22:26:51 STEP 1707 H=1707:00 +CPU= 43.038
  22:27:21 STEP 1708 H=1708:00 +CPU= 29.373
  22:27:49 STEP 1709 H=1709:00 +CPU= 28.411
  22:28:34 STEP 1710 H=1710:00 +CPU= 44.194
  22:29:02 STEP 1711 H=1711:00 +CPU= 28.320
  22:29:32 STEP 1712 H=1712:00 +CPU= 29.765
  22:30:16 STEP 1713 H=1713:00 +CPU= 43.602
  22:30:45 STEP 1714 H=1714:00 +CPU= 29.387
  22:31:14 STEP 1715 H=1715:00 +CPU= 28.410
  22:32:03 STEP 1716 H=1716:00 +CPU= 48.750
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001716
  22:32:32 STEP 1717 H=1717:00 +CPU= 28.591
  22:33:01 STEP 1718 H=1718:00 +CPU= 29.755
  22:33:46 STEP 1719 H=1719:00 +CPU= 43.859
  22:34:15 STEP 1720 H=1720:00 +CPU= 29.316
  22:34:44 STEP 1721 H=1721:00 +CPU= 28.407
  22:35:29 STEP 1722 H=1722:00 +CPU= 44.804
  22:35:57 STEP 1723 H=1723:00 +CPU= 28.588
  22:36:27 STEP 1724 H=1724:00 +CPU= 29.529
  22:37:11 STEP 1725 H=1725:00 +CPU= 43.506
  22:37:40 STEP 1726 H=1726:00 +CPU= 29.497
  22:38:10 STEP 1727 H=1727:00 +CPU= 29.717
  22:39:01 STEP 1728 H=1728:00 +CPU= 51.056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_71.zip
Uploading the intermediate file: upload_file_71.zip
  22:39:30 STEP 1729 H=1729:00 +CPU= 28.427
  22:39:59 STEP 1730 H=1730:00 +CPU= 29.193
  22:40:43 STEP 1731 H=1731:00 +CPU= 43.078
  22:41:12 STEP 1732 H=1732:00 +CPU= 29.361
  22:41:40 STEP 1733 H=1733:00 +CPU= 28.050
  22:42:25 STEP 1734 H=1734:00 +CPU= 44.661
  22:42:54 STEP 1735 H=1735:00 +CPU= 28.526
  22:43:23 STEP 1736 H=1736:00 +CPU= 29.433
  22:44:07 STEP 1737 H=1737:00 +CPU= 43.558
  22:44:36 STEP 1738 H=1738:00 +CPU= 29.094
  22:45:05 STEP 1739 H=1739:00 +CPU= 28.219
  22:45:54 STEP 1740 H=1740:00 +CPU= 48.726
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001740
  22:46:22 STEP 1741 H=1741:00 +CPU= 28.431
  22:46:51 STEP 1742 H=1742:00 +CPU= 29.002
  22:47:35 STEP 1743 H=1743:00 +CPU= 43.924
  22:48:05 STEP 1744 H=1744:00 +CPU= 29.604
  22:48:34 STEP 1745 H=1745:00 +CPU= 28.300
  22:49:19 STEP 1746 H=1746:00 +CPU= 44.827
  22:49:47 STEP 1747 H=1747:00 +CPU= 28.462
  22:50:17 STEP 1748 H=1748:00 +CPU= 29.324
  22:51:01 STEP 1749 H=1749:00 +CPU= 43.949
  22:51:30 STEP 1750 H=1750:00 +CPU= 29.057
  22:52:00 STEP 1751 H=1751:00 +CPU= 29.668
  22:52:51 STEP 1752 H=1752:00 +CPU= 51.258
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_72.zip
Uploading the intermediate file: upload_file_72.zip
  22:53:20 STEP 1753 H=1753:00 +CPU= 28.167
  22:53:49 STEP 1754 H=1754:00 +CPU= 29.087
  22:54:32 STEP 1755 H=1755:00 +CPU= 43.116
  22:55:02 STEP 1756 H=1756:00 +CPU= 29.544
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0271
(argv4) batchid: 974
(argv5) wuid: 12190915
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6841f1b36ed6c39b8a368254b1c46880 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
ic_ancil_file: ic_ancil_12190915
ifsdata_file: ifsdata_12190915
climate_data_file: clim_data_12190915
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ca417744e42b7dae45aab4d8ad39f288 to: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_dc203379185d3a89bdc7c8ac732e9cb0 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12190915.xml
-- Model is restarting --
Adjusting last_iter, 1757, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
last_cpu_time: 70263
upload_file_number: 73
last_iter: 1751
last_upload: 6307200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461
The child process has been launched with process id: 344321
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344321:344321] [20230107:230149:1673132509:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:03:16 STEP 1752 H=1752:00 +CPU= 82.326
  23:04:05 STEP 1753 H=1753:00 +CPU= 49.285
  23:04:51 STEP 1754 H=1754:00 +CPU= 45.262
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0271
(argv4) batchid: 974
(argv5) wuid: 12190915
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6841f1b36ed6c39b8a368254b1c46880 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0271_2005050100_123_974_12190915.zip
ic_ancil_file: ic_ancil_12190915
ifsdata_file: ifsdata_12190915
climate_data_file: clim_data_12190915
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ca417744e42b7dae45aab4d8ad39f288 to: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12190915.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_dc203379185d3a89bdc7c8ac732e9cb0 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12190915.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12190915.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12190915.xml
-- Model is restarting --
Adjusting last_iter, 1755, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12190915.xml
last_cpu_time: 70481
upload_file_number: 73
last_iter: 1751
last_upload: 6307200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461
The child process has been launched with process id: 344413
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:344413:344413] [20230107:230840:1673132920:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:10:29 STEP 1752 H=1752:00 +CPU= 82.732
  23:11:18 STEP 1753 H=1753:00 +CPU= 47.966
  23:12:05 STEP 1754 H=1754:00 +CPU= 46.742
  23:13:21 STEP 1755 H=1755:00 +CPU= 75.520
  23:14:14 STEP 1756 H=1756:00 +CPU= 52.783
  23:15:00 STEP 1757 H=1757:00 +CPU= 45.494
  23:16:14 STEP 1758 H=1758:00 +CPU= 74.255
  23:17:06 STEP 1759 H=1759:00 +CPU= 51.796
  23:17:54 STEP 1760 H=1760:00 +CPU= 47.834
  23:19:02 STEP 1761 H=1761:00 +CPU= 66.921
  23:19:51 STEP 1762 H=1762:00 +CPU= 49.031
  23:20:32 STEP 1763 H=1763:00 +CPU= 40.375
  23:21:53 STEP 1764 H=1764:00 +CPU= 80.802
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001764
  23:22:44 STEP 1765 H=1765:00 +CPU= 50.803
  23:23:37 STEP 1766 H=1766:00 +CPU= 52.569
  23:24:49 STEP 1767 H=1767:00 +CPU= 71.074
  23:25:30 STEP 1768 H=1768:00 +CPU= 41.534
  23:26:11 STEP 1769 H=1769:00 +CPU= 40.046
  23:27:11 STEP 1770 H=1770:00 +CPU= 60.471
  23:27:52 STEP 1771 H=1771:00 +CPU= 40.234
  23:28:38 STEP 1772 H=1772:00 +CPU= 45.598
  23:29:38 STEP 1773 H=1773:00 +CPU= 60.567
  23:30:16 STEP 1774 H=1774:00 +CPU= 36.957
  23:30:53 STEP 1775 H=1775:00 +CPU= 37.470
  23:32:23 STEP 1776 H=1776:00 +CPU= 83.644
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_73.zip
Uploading the intermediate file: upload_file_73.zip
  23:33:17 STEP 1777 H=1777:00 +CPU= 53.812
  23:34:10 STEP 1778 H=1778:00 +CPU= 52.107
  23:35:30 STEP 1779 H=1779:00 +CPU= 79.899
  23:36:20 STEP 1780 H=1780:00 +CPU= 49.667
  23:37:04 STEP 1781 H=1781:00 +CPU= 43.101
  23:38:20 STEP 1782 H=1782:00 +CPU= 76.445
  23:39:14 STEP 1783 H=1783:00 +CPU= 53.690
  23:40:11 STEP 1784 H=1784:00 +CPU= 55.900
  23:41:29 STEP 1785 H=1785:00 +CPU= 78.114
  23:42:22 STEP 1786 H=1786:00 +CPU= 52.921
  23:43:01 STEP 1787 H=1787:00 +CPU= 38.544
  23:44:08 STEP 1788 H=1788:00 +CPU= 65.986
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001788
  23:44:54 STEP 1789 H=1789:00 +CPU= 46.696
  23:45:40 STEP 1790 H=1790:00 +CPU= 44.826
  23:46:38 STEP 1791 H=1791:00 +CPU= 58.071
  23:47:15 STEP 1792 H=1792:00 +CPU= 37.346
  23:47:52 STEP 1793 H=1793:00 +CPU= 36.238
  23:48:47 STEP 1794 H=1794:00 +CPU= 55.099
  23:49:26 STEP 1795 H=1795:00 +CPU= 38.597
  23:50:03 STEP 1796 H=1796:00 +CPU= 37.194
  23:50:58 STEP 1797 H=1797:00 +CPU= 54.505
  23:51:36 STEP 1798 H=1798:00 +CPU= 37.576
  23:52:12 STEP 1799 H=1799:00 +CPU= 35.569
  23:53:27 STEP 1800 H=1800:00 +CPU= 71.794
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_74.zip
Uploading the intermediate file: upload_file_74.zip
  23:54:05 STEP 1801 H=1801:00 +CPU= 37.847
  23:54:41 STEP 1802 H=1802:00 +CPU= 35.254
  23:55:34 STEP 1803 H=1803:00 +CPU= 53.186
  23:56:04 STEP 1804 H=1804:00 +CPU= 29.636
  23:56:32 STEP 1805 H=1805:00 +CPU= 28.071
  23:57:17 STEP 1806 H=1806:00 +CPU= 44.213
  23:57:45 STEP 1807 H=1807:00 +CPU= 28.176
  23:58:14 STEP 1808 H=1808:00 +CPU= 28.892
  23:58:57 STEP 1809 H=1809:00 +CPU= 43.115
  23:59:26 STEP 1810 H=1810:00 +CPU= 28.733
  23:59:54 STEP 1811 H=1811:00 +CPU= 27.687
  00:00:42 STEP 1812 H=1812:00 +CPU= 47.748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001812
  00:01:10 STEP 1813 H=1813:00 +CPU= 28.182
  00:01:39 STEP 1814 H=1814:00 +CPU= 28.526
  00:02:22 STEP 1815 H=1815:00 +CPU= 43.205
  00:02:51 STEP 1816 H=1816:00 +CPU= 28.726
  00:03:19 STEP 1817 H=1817:00 +CPU= 27.788
  00:04:04 STEP 1818 H=1818:00 +CPU= 44.253
  00:04:32 STEP 1819 H=1819:00 +CPU= 27.922
  00:05:00 STEP 1820 H=1820:00 +CPU= 28.487
  00:05:44 STEP 1821 H=1821:00 +CPU= 43.226
  00:06:13 STEP 1822 H=1822:00 +CPU= 28.868
  00:06:41 STEP 1823 H=1823:00 +CPU= 28.240
  00:07:35 STEP 1824 H=1824:00 +CPU= 50.432
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_75.zip
Uploading the intermediate file: upload_file_75.zip
  00:08:04 STEP 1825 H=1825:00 +CPU= 28.091
  00:08:32 STEP 1826 H=1826:00 +CPU= 28.540
  00:09:16 STEP 1827 H=1827:00 +CPU= 43.385
  00:09:45 STEP 1828 H=1828:00 +CPU= 28.840
  00:10:13 STEP 1829 H=1829:00 +CPU= 27.783
  00:10:57 STEP 1830 H=1830:00 +CPU= 43.595
  00:11:25 STEP 1831 H=1831:00 +CPU= 27.947
  00:11:53 STEP 1832 H=1832:00 +CPU= 28.511
  00:12:37 STEP 1833 H=1833:00 +CPU= 43.140
  00:13:05 STEP 1834 H=1834:00 +CPU= 28.568
  00:13:33 STEP 1835 H=1835:00 +CPU= 27.993
  00:14:21 STEP 1836 H=1836:00 +CPU= 47.551
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001836
  00:14:49 STEP 1837 H=1837:00 +CPU= 27.928
  00:15:18 STEP 1838 H=1838:00 +CPU= 28.895
  00:16:01 STEP 1839 H=1839:00 +CPU= 42.640
  00:16:30 STEP 1840 H=1840:00 +CPU= 28.718
  00:16:57 STEP 1841 H=1841:00 +CPU= 27.430
  00:17:42 STEP 1842 H=1842:00 +CPU= 44.369
  00:18:10 STEP 1843 H=1843:00 +CPU= 27.932
  00:18:38 STEP 1844 H=1844:00 +CPU= 28.113
  00:19:22 STEP 1845 H=1845:00 +CPU= 43.340
  00:19:51 STEP 1846 H=1846:00 +CPU= 28.758
  00:20:19 STEP 1847 H=1847:00 +CPU= 27.700
  00:21:13 STEP 1848 H=1848:00 +CPU= 50.160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190915/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0271_2005050100_123_974_12190915_1_r429778461_76.zip
Uploading the intermediate file: upload_file_76.zip
  00:21:41 STEP 1849 H=1849:00 +CPU= 27.329
  00:22:09 STEP 1850 H=1850:00 +CPU= 28.542
  00:22:52 STEP 1851 H=1851:00 +CPU= 42.778
  00:23:21 STEP 1852 H=1852:00 +CPU= 28.745
  00:23:49 STEP 1853 H=1853:00 +CPU= 27.596
  00:24:33 STEP 1854 H=1854:00 +CPU= 43.750
  00:25:01 STEP 1855 H=1855:00 +CPU= 27.680
  00:25:29 STEP 1856 H=1856:00 +CPU= 28.622
  00:26:13 STEP 1857 H=1857:00 +CPU= 43.002
  00:26:41 STEP 1858 H=1858:00 +CPU= 28.629
  00:27:09 STEP 1859 H=1859:00 +CPU= 27.513
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:01 1491267 22293431 oifs_43r3_ps_0271_2005050100_123_974_12190915_1 5,443,200 59,552 0.0109
09 May 2023 16:11:53 1491267 22293431 oifs_43r3_ps_0271_2005050100_123_974_12190915_1 4,579,200 50,138 0.0109
09 May 2023 16:11:27 1491267 22293431 oifs_43r3_ps_0271_2005050100_123_974_12190915_1 2,505,600 25,873 0.0103
09 May 2023 16:11:07 1491267 22293431 oifs_43r3_ps_0271_2005050100_123_974_12190915_1 864,000 10,187 0.0118


©2024 climateprediction.net