climateprediction.net home page
Task 22306789

Task 22306789

Name oifs_43r3_ps_0283_2000050100_123_969_12185927_2
Workunit 12185927
Created 9 Feb 2023, 11:22:14 UTC
Sent 9 Feb 2023, 11:22:42 UTC
Report deadline 10 Apr 2023, 11:22:42 UTC
Received 12 Feb 2023, 3:29:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1332501
Run time 2 days 4 hours 51 min 32 sec
CPU time 17 hours 42 min 59 sec
Validate state Invalid
Credit 5,080.32
Device peak FLOPS 2.89 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,404.64 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,674.48 MB
Stderr 
<core_client_version>7.6.33</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded
</message>
<stderr_txt>
for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_80c745e61caa1f276186000483c3d9f3 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
ic_ancil_file: ic_ancil_12185927
ifsdata_file: ifsdata_12185927
climate_data_file: clim_data_12185927
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d131828f2e917bf22044ebe911f153cc to: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04167562849a5896b6063dee34e58e1c to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185927.xml
-- Model is restarting --
Adjusting last_iter, 1086, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
last_cpu_time: 59206
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459
The child process has been launched with process id: 29106
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.029] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:29106:29106] [20230211:124740:1676141260:0.030] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:55:40 STEP 1080 H=1080:00 +CPU= 61.116
  12:57:51 STEP 1081 H=1081:00 +CPU= 33.600
  12:59:49 STEP 1082 H=1082:00 +CPU= 34.304
  13:00:43 STEP 1083 H=1083:00 +CPU= 51.984
  13:01:16 STEP 1084 H=1084:00 +CPU= 33.104
  13:03:00 STEP 1085 H=1085:00 +CPU= 33.672
  13:04:03 STEP 1086 H=1086:00 +CPU= 52.240
  13:05:22 STEP 1087 H=1087:00 +CPU= 33.048
  13:05:57 STEP 1088 H=1088:00 +CPU= 33.356
  13:06:51 STEP 1089 H=1089:00 +CPU= 51.420
  13:08:58 STEP 1090 H=1090:00 +CPU= 35.336
  13:09:38 STEP 1091 H=1091:00 +CPU= 32.156
  14:05:52 STEP 1092 H=1092:00 +CPU= 99.384
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001092
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0283
(argv4) batchid: 969
(argv5) wuid: 12185927
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_80c745e61caa1f276186000483c3d9f3 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
ic_ancil_file: ic_ancil_12185927
ifsdata_file: ifsdata_12185927
climate_data_file: clim_data_12185927
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d131828f2e917bf22044ebe911f153cc to: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04167562849a5896b6063dee34e58e1c to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185927.xml
-- Model is restarting --
Adjusting last_iter, 1093, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
last_cpu_time: 59797
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459
The child process has been launched with process id: 2591
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:2591:2591] [20230211:141139:1676146299:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:14:47 STEP 1080 H=1080:00 +CPU= 59.444
  14:15:21 STEP 1081 H=1081:00 +CPU= 32.432
  14:17:50 STEP 1082 H=1082:00 +CPU= 35.092
  14:20:04 STEP 1083 H=1083:00 +CPU= 52.876
  14:20:38 STEP 1084 H=1084:00 +CPU= 33.272
  14:22:15 STEP 1085 H=1085:00 +CPU= 33.008
  14:23:13 STEP 1086 H=1086:00 +CPU= 52.476
  14:23:45 STEP 1087 H=1087:00 +CPU= 32.140
  14:24:18 STEP 1088 H=1088:00 +CPU= 33.276
  15:20:48 STEP 1089 H=1089:00 +CPU= 76.656
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0283
(argv4) batchid: 969
(argv5) wuid: 12185927
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_80c745e61caa1f276186000483c3d9f3 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
ic_ancil_file: ic_ancil_12185927
ifsdata_file: ifsdata_12185927
climate_data_file: clim_data_12185927
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d131828f2e917bf22044ebe911f153cc to: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04167562849a5896b6063dee34e58e1c to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185927.xml
-- Model is restarting --
Adjusting last_iter, 1090, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
last_cpu_time: 60253
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459
The child process has been launched with process id: 7234
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:7234:7234] [20230211:153405:1676151245:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:37:34 STEP 1080 H=1080:00 +CPU= 60.316
  15:38:08 STEP 1081 H=1081:00 +CPU= 32.432
  15:50:55 STEP 1082 H=1082:00 +CPU= 59.348
  16:04:24 STEP 1083 H=1083:00 +CPU= 65.360
  16:19:56 STEP 1084 H=1084:00 +CPU= 53.416
  16:43:26 STEP 1085 H=1085:00 +CPU= 45.600
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0283
(argv4) batchid: 969
(argv5) wuid: 12185927
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_80c745e61caa1f276186000483c3d9f3 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
ic_ancil_file: ic_ancil_12185927
ifsdata_file: ifsdata_12185927
climate_data_file: clim_data_12185927
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d131828f2e917bf22044ebe911f153cc to: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04167562849a5896b6063dee34e58e1c to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185927.xml
-- Model is restarting --
Adjusting last_iter, 1086, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
last_cpu_time: 60601
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459
The child process has been launched with process id: 12461
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:12461:12461] [20230211:164730:1676155650:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:50:23 STEP 1080 H=1080:00 +CPU= 58.724
  16:51:36 STEP 1081 H=1081:00 +CPU= 33.540
  16:52:11 STEP 1082 H=1082:00 +CPU= 32.928
  16:53:05 STEP 1083 H=1083:00 +CPU= 51.144
  16:53:39 STEP 1084 H=1084:00 +CPU= 33.896
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0283
(argv4) batchid: 969
(argv5) wuid: 12185927
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_80c745e61caa1f276186000483c3d9f3 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
ic_ancil_file: ic_ancil_12185927
ifsdata_file: ifsdata_12185927
climate_data_file: clim_data_12185927
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d131828f2e917bf22044ebe911f153cc to: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04167562849a5896b6063dee34e58e1c to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185927.xml
-- Model is restarting --
Adjusting last_iter, 1085, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
last_cpu_time: 60887
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459
The child process has been launched with process id: 18254
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:18254:18254] [20230211:175005:1676159405:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:53:56 STEP 1080 H=1080:00 +CPU= 61.056
  17:54:30 STEP 1081 H=1081:00 +CPU= 32.116
  17:55:03 STEP 1082 H=1082:00 +CPU= 33.132
  17:55:55 STEP 1083 H=1083:00 +CPU= 51.192
  17:56:29 STEP 1084 H=1084:00 +CPU= 33.024
  17:57:10 STEP 1085 H=1085:00 +CPU= 32.860
  17:58:11 STEP 1086 H=1086:00 +CPU= 53.020
  17:58:45 STEP 1087 H=1087:00 +CPU= 32.916
  17:59:23 STEP 1088 H=1088:00 +CPU= 33.436
  18:00:15 STEP 1089 H=1089:00 +CPU= 51.384
  18:00:48 STEP 1090 H=1090:00 +CPU= 32.916
  18:01:20 STEP 1091 H=1091:00 +CPU= 31.928
  18:02:19 STEP 1092 H=1092:00 +CPU= 56.500
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001092
  18:02:51 STEP 1093 H=1093:00 +CPU= 31.932
  18:03:24 STEP 1094 H=1094:00 +CPU= 32.924
  18:04:15 STEP 1095 H=1095:00 +CPU= 50.900
  18:04:48 STEP 1096 H=1096:00 +CPU= 32.844
  18:05:20 STEP 1097 H=1097:00 +CPU= 31.848
  18:06:12 STEP 1098 H=1098:00 +CPU= 51.916
  18:06:44 STEP 1099 H=1099:00 +CPU= 31.764
  18:07:16 STEP 1100 H=1100:00 +CPU= 32.684
  18:08:07 STEP 1101 H=1101:00 +CPU= 50.636
  18:08:40 STEP 1102 H=1102:00 +CPU= 32.636
  18:09:15 STEP 1103 H=1103:00 +CPU= 32.708
  18:10:42 STEP 1104 H=1104:00 +CPU= 60.772
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459_45.zip
Uploading the intermediate file: upload_file_45.zip
  18:12:36 STEP 1105 H=1105:00 +CPU= 33.288
  18:13:10 STEP 1106 H=1106:00 +CPU= 33.304
  18:14:02 STEP 1107 H=1107:00 +CPU= 50.888
  18:14:35 STEP 1108 H=1108:00 +CPU= 32.892
  18:15:07 STEP 1109 H=1109:00 +CPU= 31.908
  18:15:59 STEP 1110 H=1110:00 +CPU= 51.980
  18:16:31 STEP 1111 H=1111:00 +CPU= 31.908
  18:17:04 STEP 1112 H=1112:00 +CPU= 32.932
  18:17:55 STEP 1113 H=1113:00 +CPU= 50.920
  18:18:28 STEP 1114 H=1114:00 +CPU= 32.936
  18:19:00 STEP 1115 H=1115:00 +CPU= 31.960
  18:20:00 STEP 1116 H=1116:00 +CPU= 56.496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001116
  18:20:32 STEP 1117 H=1117:00 +CPU= 31.996
  18:21:05 STEP 1118 H=1118:00 +CPU= 32.928
  18:21:56 STEP 1119 H=1119:00 +CPU= 50.852
  18:22:29 STEP 1120 H=1120:00 +CPU= 33.056
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0283
(argv4) batchid: 969
(argv5) wuid: 12185927
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_80c745e61caa1f276186000483c3d9f3 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0283_2000050100_123_969_12185927.zip
ic_ancil_file: ic_ancil_12185927
ifsdata_file: ifsdata_12185927
climate_data_file: clim_data_12185927
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d131828f2e917bf22044ebe911f153cc to: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12185927.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04167562849a5896b6063dee34e58e1c to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12185927.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12185927.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12185927.xml
-- Model is restarting --
Adjusting last_iter, 1121, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12185927.xml
last_cpu_time: 62604
upload_file_number: 46
last_iter: 1103
last_upload: 3974400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459
The child process has been launched with process id: 26253
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:nemo:1:1:26253:26253] [20230211:192715:1676165235:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:31:36 STEP 1104 H=1104:00 +CPU= 61.360
  19:32:08 STEP 1105 H=1105:00 +CPU= 32.088
  19:34:25 STEP 1106 H=1106:00 +CPU= 33.644
  19:36:38 STEP 1107 H=1107:00 +CPU= 52.484
  19:37:13 STEP 1108 H=1108:00 +CPU= 33.232
  19:37:45 STEP 1109 H=1109:00 +CPU= 32.180
  20:38:39 STEP 1110 H=1110:00 +CPU= 98.128
  20:55:32 STEP 1111 H=1111:00 +CPU= 59.724
  20:57:09 STEP 1112 H=1112:00 +CPU= 35.024
  20:58:09 STEP 1113 H=1113:00 +CPU= 54.264
  20:58:44 STEP 1114 H=1114:00 +CPU= 34.836
  20:59:18 STEP 1115 H=1115:00 +CPU= 33.976
  21:00:21 STEP 1116 H=1116:00 +CPU= 59.208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001116
  21:00:55 STEP 1117 H=1117:00 +CPU= 33.836
  21:01:30 STEP 1118 H=1118:00 +CPU= 34.748
  21:02:23 STEP 1119 H=1119:00 +CPU= 53.452
  21:02:58 STEP 1120 H=1120:00 +CPU= 34.800
  21:03:32 STEP 1121 H=1121:00 +CPU= 33.892
  21:04:28 STEP 1122 H=1122:00 +CPU= 54.792
  21:05:02 STEP 1123 H=1123:00 +CPU= 33.832
  21:05:37 STEP 1124 H=1124:00 +CPU= 34.836
  21:06:31 STEP 1125 H=1125:00 +CPU= 53.360
  21:07:06 STEP 1126 H=1126:00 +CPU= 34.764
  21:07:42 STEP 1127 H=1127:00 +CPU= 34.904
  21:08:58 STEP 1128 H=1128:00 +CPU= 60.560
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12185927/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0283_2000050100_123_969_12185927_2_r1140877459_46.zip
Uploading the intermediate file: upload_file_46.zip
  21:09:32 STEP 1129 H=1129:00 +CPU= 33.872
oifs_43r3_ps_1.05_x86_64-pc-linux-gnu: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:50:37 1332501 22306789 oifs_43r3_ps_0283_2000050100_123_969_12185927_2 3,628,800 55,861 0.0154
09 May 2023 16:50:36 1332501 22306789 oifs_43r3_ps_0283_2000050100_123_969_12185927_2 2,764,800 45,916 0.0166
09 May 2023 16:50:10 1332501 22306789 oifs_43r3_ps_0283_2000050100_123_969_12185927_2 864,000 9,367 0.0108


©2024 cpdn.org