climateprediction.net home page
Task 22247284

Task 22247284

Name oifs_43r3_ps_2179_2021050100_123_947_12165268_0
Workunit 12165268
Created 29 Nov 2022, 13:18:27 UTC
Sent 29 Nov 2022, 20:52:03 UTC
Report deadline 29 Dec 2022, 20:52:03 UTC
Received 26 Dec 2022, 11:28:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1520117
Run time 1 days 22 hours 57 min 50 sec
CPU time 30 min 28 sec
Validate state Invalid
Credit 11,128.32
Device peak FLOPS 3.37 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,419.17 MB
Peak swap size 4,978.08 MB
Peak disk usage 38,490.57 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
hook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:62064:62064] [20221225:235104:1672033864:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:62064:62064] [20221225:235104:1672033864:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:52:48 STEP 2016 H=2016:00 +CPU= 98.428
  23:53:44 STEP 2017 H=2017:00 +CPU= 55.634
  23:54:39 STEP 2018 H=2018:00 +CPU= 54.707
  23:56:01 STEP 2019 H=2019:00 +CPU= 81.328
  23:56:54 STEP 2020 H=2020:00 +CPU= 53.146
  23:57:47 STEP 2021 H=2021:00 +CPU= 52.304
  23:59:12 STEP 2022 H=2022:00 +CPU= 84.255
  00:00:04 STEP 2023 H=2023:00 +CPU= 51.844
  00:01:00 STEP 2024 H=2024:00 +CPU= 55.382
  00:02:24 STEP 2025 H=2025:00 +CPU= 83.627
  00:03:18 STEP 2026 H=2026:00 +CPU= 52.899
  00:04:09 STEP 2027 H=2027:00 +CPU= 51.262
  00:05:42 STEP 2028 H=2028:00 +CPU= 92.168
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002028
  00:06:36 STEP 2029 H=2029:00 +CPU= 52.595
  00:07:31 STEP 2030 H=2030:00 +CPU= 54.559
  00:08:54 STEP 2031 H=2031:00 +CPU= 82.559
  00:09:48 STEP 2032 H=2032:00 +CPU= 53.304
  00:10:42 STEP 2033 H=2033:00 +CPU= 53.127
  00:12:13 STEP 2034 H=2034:00 +CPU= 90.195
  00:13:04 STEP 2035 H=2035:00 +CPU= 51.461
  00:14:00 STEP 2036 H=2036:00 +CPU= 55.304
  00:15:23 STEP 2037 H=2037:00 +CPU= 81.858
  00:16:20 STEP 2038 H=2038:00 +CPU= 56.441
  00:17:16 STEP 2039 H=2039:00 +CPU= 55.513
  00:18:58 STEP 2040 H=2040:00 +CPU=101.486
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7344000
  00:19:54 STEP 2041 H=2041:00 +CPU= 55.822
  00:20:50 STEP 2042 H=2042:00 +CPU= 55.176
  00:22:16 STEP 2043 H=2043:00 +CPU= 85.999
  00:23:15 STEP 2044 H=2044:00 +CPU= 58.006
  00:24:09 STEP 2045 H=2045:00 +CPU= 53.555
  00:25:39 STEP 2046 H=2046:00 +CPU= 90.020
  00:26:35 STEP 2047 H=2047:00 +CPU= 55.898
  00:27:31 STEP 2048 H=2048:00 +CPU= 55.130
  00:28:57 STEP 2049 H=2049:00 +CPU= 86.195
  00:29:54 STEP 2050 H=2050:00 +CPU= 55.802
  00:30:46 STEP 2051 H=2051:00 +CPU= 51.856
  00:32:22 STEP 2052 H=2052:00 +CPU= 94.368
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002052
  00:33:18 STEP 2053 H=2053:00 +CPU= 52.685
  00:34:14 STEP 2054 H=2054:00 +CPU= 55.866
  00:35:41 STEP 2055 H=2055:00 +CPU= 86.222
  00:36:38 STEP 2056 H=2056:00 +CPU= 56.700
  00:37:33 STEP 2057 H=2057:00 +CPU= 54.630
  00:39:03 STEP 2058 H=2058:00 +CPU= 89.285
  00:39:59 STEP 2059 H=2059:00 +CPU= 56.345
  00:40:55 STEP 2060 H=2060:00 +CPU= 54.927
  00:42:16 STEP 2061 H=2061:00 +CPU= 80.614
  00:43:14 STEP 2062 H=2062:00 +CPU= 57.988
  00:44:12 STEP 2063 H=2063:00 +CPU= 57.679
  00:45:54 STEP 2064 H=2064:00 +CPU=101.704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 7430400
  00:46:48 STEP 2065 H=2065:00 +CPU= 52.620
  00:47:41 STEP 2066 H=2066:00 +CPU= 53.444
  00:49:08 STEP 2067 H=2067:00 +CPU= 85.812
  00:50:05 STEP 2068 H=2068:00 +CPU= 57.427
  00:50:59 STEP 2069 H=2069:00 +CPU= 53.022
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2179
(argv4) batchid: 947
(argv5) wuid: 12165268
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7386c705b8dc7a0ad901bf4a74d531f7 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
ic_ancil_file: ic_ancil_12165268
ifsdata_file: ifsdata_12165268
climate_data_file: clim_data_12165268
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_1301d26c6254c9acd665a99f46937aa2 to: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3129d7ba57ce52197195a1c4b6eb5b40 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287
The child process has been launched with process id: 63200
Executing the command: /var/lib/boinc-client/slots/1/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63200:63200] [20221226:005214:1672037534:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:53:56 STEP 2064 H=2064:00 +CPU= 96.612
  00:54:50 STEP 2065 H=2065:00 +CPU= 53.316
  00:55:47 STEP 2066 H=2066:00 +CPU= 56.945
  00:57:10 STEP 2067 H=2067:00 +CPU= 82.161
  00:58:04 STEP 2068 H=2068:00 +CPU= 53.460
  00:58:56 STEP 2069 H=2069:00 +CPU= 51.544
  01:00:20 STEP 2070 H=2070:00 +CPU= 83.958
  01:01:12 STEP 2071 H=2071:00 +CPU= 51.349
  01:02:08 STEP 2072 H=2072:00 +CPU= 56.010
  01:03:37 STEP 2073 H=2073:00 +CPU= 87.998
  01:04:33 STEP 2074 H=2074:00 +CPU= 55.517
  01:05:27 STEP 2075 H=2075:00 +CPU= 54.364
  01:07:07 STEP 2076 H=2076:00 +CPU= 98.540
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002076
  01:08:01 STEP 2077 H=2077:00 +CPU= 54.358
  01:08:58 STEP 2078 H=2078:00 +CPU= 56.258
  01:10:25 STEP 2079 H=2079:00 +CPU= 86.986
  01:11:21 STEP 2080 H=2080:00 +CPU= 54.707
  01:12:15 STEP 2081 H=2081:00 +CPU= 54.125
  01:13:46 STEP 2082 H=2082:00 +CPU= 89.992
  01:14:40 STEP 2083 H=2083:00 +CPU= 53.996
  01:15:36 STEP 2084 H=2084:00 +CPU= 55.125
  01:17:04 STEP 2085 H=2085:00 +CPU= 87.477
  01:17:59 STEP 2086 H=2086:00 +CPU= 54.983
  01:18:55 STEP 2087 H=2087:00 +CPU= 55.907
  01:20:40 STEP 2088 H=2088:00 +CPU=103.736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7516800
  01:21:35 STEP 2089 H=2089:00 +CPU= 54.843
  01:22:32 STEP 2090 H=2090:00 +CPU= 56.225
  01:23:57 STEP 2091 H=2091:00 +CPU= 84.221
  01:24:52 STEP 2092 H=2092:00 +CPU= 55.012
  01:25:47 STEP 2093 H=2093:00 +CPU= 54.618
  01:27:18 STEP 2094 H=2094:00 +CPU= 90.489
  01:28:12 STEP 2095 H=2095:00 +CPU= 53.385
  01:29:09 STEP 2096 H=2096:00 +CPU= 56.381
  01:30:37 STEP 2097 H=2097:00 +CPU= 88.530
  01:31:34 STEP 2098 H=2098:00 +CPU= 56.168
  01:32:29 STEP 2099 H=2099:00 +CPU= 54.789
  01:34:06 STEP 2100 H=2100:00 +CPU= 96.741
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002100
  01:35:00 STEP 2101 H=2101:00 +CPU= 53.423
  01:35:57 STEP 2102 H=2102:00 +CPU= 56.228
  01:37:22 STEP 2103 H=2103:00 +CPU= 84.163
  01:38:16 STEP 2104 H=2104:00 +CPU= 53.455
  01:39:11 STEP 2105 H=2105:00 +CPU= 54.926
  01:40:38 STEP 2106 H=2106:00 +CPU= 85.992
  01:41:32 STEP 2107 H=2107:00 +CPU= 54.283
  01:42:28 STEP 2108 H=2108:00 +CPU= 55.065
  01:43:53 STEP 2109 H=2109:00 +CPU= 85.359
  01:44:46 STEP 2110 H=2110:00 +CPU= 51.767
  01:45:41 STEP 2111 H=2111:00 +CPU= 55.079
  01:47:23 STEP 2112 H=2112:00 +CPU=100.369
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 7603200
  01:48:19 STEP 2113 H=2113:00 +CPU= 55.739
  01:49:15 STEP 2114 H=2114:00 +CPU= 55.608
  01:50:41 STEP 2115 H=2115:00 +CPU= 86.212
  01:51:37 STEP 2116 H=2116:00 +CPU= 55.766
  01:52:31 STEP 2117 H=2117:00 +CPU= 53.295
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2179
(argv4) batchid: 947
(argv5) wuid: 12165268
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7386c705b8dc7a0ad901bf4a74d531f7 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
ic_ancil_file: ic_ancil_12165268
ifsdata_file: ifsdata_12165268
climate_data_file: clim_data_12165268
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_1301d26c6254c9acd665a99f46937aa2 to: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3129d7ba57ce52197195a1c4b6eb5b40 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287
The child process has been launched with process id: 63733
Executing the command: /var/lib/boinc-client/slots/1/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63733:63733] [20221226:015355:1672041235:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:55:39 STEP 2112 H=2112:00 +CPU= 98.205
  01:56:34 STEP 2113 H=2113:00 +CPU= 54.356
  01:57:30 STEP 2114 H=2114:00 +CPU= 55.485
  01:58:57 STEP 2115 H=2115:00 +CPU= 86.375
  01:59:51 STEP 2116 H=2116:00 +CPU= 54.080
  02:00:47 STEP 2117 H=2117:00 +CPU= 55.549
  02:02:15 STEP 2118 H=2118:00 +CPU= 88.001
  02:03:09 STEP 2119 H=2119:00 +CPU= 53.487
  02:04:04 STEP 2120 H=2120:00 +CPU= 54.272
  02:05:31 STEP 2121 H=2121:00 +CPU= 86.928
  02:06:27 STEP 2122 H=2122:00 +CPU= 55.571
  02:07:23 STEP 2123 H=2123:00 +CPU= 55.061
  02:08:59 STEP 2124 H=2124:00 +CPU= 95.854
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002124
  02:09:54 STEP 2125 H=2125:00 +CPU= 54.467
  02:10:49 STEP 2126 H=2126:00 +CPU= 54.534
  02:12:16 STEP 2127 H=2127:00 +CPU= 86.544
  02:13:12 STEP 2128 H=2128:00 +CPU= 55.225
  02:14:07 STEP 2129 H=2129:00 +CPU= 54.381
  02:15:36 STEP 2130 H=2130:00 +CPU= 88.163
  02:16:31 STEP 2131 H=2131:00 +CPU= 55.008
  02:17:27 STEP 2132 H=2132:00 +CPU= 55.573
  02:18:54 STEP 2133 H=2133:00 +CPU= 86.689
  02:19:51 STEP 2134 H=2134:00 +CPU= 56.842
  02:20:48 STEP 2135 H=2135:00 +CPU= 56.294
  02:22:31 STEP 2136 H=2136:00 +CPU=102.553
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7689600
  02:23:26 STEP 2137 H=2137:00 +CPU= 55.048
  02:24:23 STEP 2138 H=2138:00 +CPU= 55.699
  02:25:50 STEP 2139 H=2139:00 +CPU= 86.288
  02:26:45 STEP 2140 H=2140:00 +CPU= 55.564
  02:27:38 STEP 2141 H=2141:00 +CPU= 52.619
  02:29:08 STEP 2142 H=2142:00 +CPU= 89.146
  02:30:03 STEP 2143 H=2143:00 +CPU= 55.057
  02:30:58 STEP 2144 H=2144:00 +CPU= 54.354
  02:32:25 STEP 2145 H=2145:00 +CPU= 86.650
  02:33:22 STEP 2146 H=2146:00 +CPU= 56.131
  02:34:16 STEP 2147 H=2147:00 +CPU= 54.350
  02:35:55 STEP 2148 H=2148:00 +CPU= 97.736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002148
  02:36:51 STEP 2149 H=2149:00 +CPU= 55.792
  02:37:46 STEP 2150 H=2150:00 +CPU= 55.331
  02:39:15 STEP 2151 H=2151:00 +CPU= 87.721
  02:40:13 STEP 2152 H=2152:00 +CPU= 58.140
  02:41:07 STEP 2153 H=2153:00 +CPU= 52.985
  02:42:36 STEP 2154 H=2154:00 +CPU= 88.637
  02:43:31 STEP 2155 H=2155:00 +CPU= 54.235
  02:44:28 STEP 2156 H=2156:00 +CPU= 56.447
  02:45:55 STEP 2157 H=2157:00 +CPU= 86.523
  02:46:49 STEP 2158 H=2158:00 +CPU= 54.075
  02:47:47 STEP 2159 H=2159:00 +CPU= 58.149
  02:49:28 STEP 2160 H=2160:00 +CPU= 99.908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 7776000
  02:50:21 STEP 2161 H=2161:00 +CPU= 52.496
  02:51:18 STEP 2162 H=2162:00 +CPU= 56.214
  02:52:43 STEP 2163 H=2163:00 +CPU= 84.479
  02:53:35 STEP 2164 H=2164:00 +CPU= 52.009
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2179
(argv4) batchid: 947
(argv5) wuid: 12165268
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7386c705b8dc7a0ad901bf4a74d531f7 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
ic_ancil_file: ic_ancil_12165268
ifsdata_file: ifsdata_12165268
climate_data_file: clim_data_12165268
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_1301d26c6254c9acd665a99f46937aa2 to: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3129d7ba57ce52197195a1c4b6eb5b40 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287
The child process has been launched with process id: 64286
Executing the command: /var/lib/boinc-client/slots/1/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64286:64286] [20221226:025518:1672044918:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:57:02 STEP 2160 H=2160:00 +CPU= 98.222
  02:57:58 STEP 2161 H=2161:00 +CPU= 55.549
  02:58:54 STEP 2162 H=2162:00 +CPU= 55.639
  03:00:21 STEP 2163 H=2163:00 +CPU= 86.957
  03:01:19 STEP 2164 H=2164:00 +CPU= 56.900
  03:02:14 STEP 2165 H=2165:00 +CPU= 55.142
  03:03:44 STEP 2166 H=2166:00 +CPU= 89.351
  03:04:39 STEP 2167 H=2167:00 +CPU= 55.361
  03:05:36 STEP 2168 H=2168:00 +CPU= 55.848
  03:07:04 STEP 2169 H=2169:00 +CPU= 87.549
  03:08:01 STEP 2170 H=2170:00 +CPU= 56.877
  03:08:55 STEP 2171 H=2171:00 +CPU= 53.255
  03:10:30 STEP 2172 H=2172:00 +CPU= 94.477
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002172
  03:11:24 STEP 2173 H=2173:00 +CPU= 53.791
  03:12:19 STEP 2174 H=2174:00 +CPU= 54.501
  03:13:49 STEP 2175 H=2175:00 +CPU= 89.059
  03:14:46 STEP 2176 H=2176:00 +CPU= 56.966
  03:15:40 STEP 2177 H=2177:00 +CPU= 53.580
  03:17:11 STEP 2178 H=2178:00 +CPU= 90.567
  03:18:07 STEP 2179 H=2179:00 +CPU= 55.486
  03:19:02 STEP 2180 H=2180:00 +CPU= 54.905
  03:20:28 STEP 2181 H=2181:00 +CPU= 85.658
  03:21:25 STEP 2182 H=2182:00 +CPU= 56.363
  03:22:21 STEP 2183 H=2183:00 +CPU= 55.610
  03:24:01 STEP 2184 H=2184:00 +CPU= 99.439
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7862400
  03:24:56 STEP 2185 H=2185:00 +CPU= 54.949
  03:25:52 STEP 2186 H=2186:00 +CPU= 55.286
  03:27:22 STEP 2187 H=2187:00 +CPU= 89.198
  03:28:21 STEP 2188 H=2188:00 +CPU= 58.505
  03:29:14 STEP 2189 H=2189:00 +CPU= 53.114
  03:30:41 STEP 2190 H=2190:00 +CPU= 86.336
  03:31:38 STEP 2191 H=2191:00 +CPU= 56.779
  03:32:34 STEP 2192 H=2192:00 +CPU= 55.524
  03:33:58 STEP 2193 H=2193:00 +CPU= 83.575
  03:34:53 STEP 2194 H=2194:00 +CPU= 54.867
  03:35:48 STEP 2195 H=2195:00 +CPU= 54.064
  03:37:26 STEP 2196 H=2196:00 +CPU= 97.558
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002196
  03:38:22 STEP 2197 H=2197:00 +CPU= 55.006
  03:39:18 STEP 2198 H=2198:00 +CPU= 56.073
  03:40:45 STEP 2199 H=2199:00 +CPU= 86.756
  03:41:43 STEP 2200 H=2200:00 +CPU= 57.214
  03:42:35 STEP 2201 H=2201:00 +CPU= 52.252
  03:44:07 STEP 2202 H=2202:00 +CPU= 91.043
  03:45:03 STEP 2203 H=2203:00 +CPU= 56.135
  03:45:59 STEP 2204 H=2204:00 +CPU= 55.120
  03:47:27 STEP 2205 H=2205:00 +CPU= 87.301
  03:48:19 STEP 2206 H=2206:00 +CPU= 52.206
  03:49:14 STEP 2207 H=2207:00 +CPU= 54.341
  03:50:53 STEP 2208 H=2208:00 +CPU= 98.764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 7948800
  03:51:48 STEP 2209 H=2209:00 +CPU= 54.422
  03:52:44 STEP 2210 H=2210:00 +CPU= 55.735
  03:54:12 STEP 2211 H=2211:00 +CPU= 87.494
  03:55:08 STEP 2212 H=2212:00 +CPU= 55.723
  03:56:02 STEP 2213 H=2213:00 +CPU= 52.948
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2179
(argv4) batchid: 947
(argv5) wuid: 12165268
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7386c705b8dc7a0ad901bf4a74d531f7 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_2179_2021050100_123_947_12165268.zip
ic_ancil_file: ic_ancil_12165268
ifsdata_file: ifsdata_12165268
climate_data_file: clim_data_12165268
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_1301d26c6254c9acd665a99f46937aa2 to: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12165268.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3129d7ba57ce52197195a1c4b6eb5b40 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12165268.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12165268.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287
The child process has been launched with process id: 64852
Executing the command: /var/lib/boinc-client/slots/1/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64852:64852] [20221226:035717:1672048637:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:59:00 STEP 2208 H=2208:00 +CPU= 97.140
  03:59:52 STEP 2209 H=2209:00 +CPU= 51.316
  04:00:51 STEP 2210 H=2210:00 +CPU= 59.018
  04:02:12 STEP 2211 H=2211:00 +CPU= 80.644
  04:03:06 STEP 2212 H=2212:00 +CPU= 53.408
  04:03:59 STEP 2213 H=2213:00 +CPU= 52.463
  04:05:22 STEP 2214 H=2214:00 +CPU= 82.543
  04:06:14 STEP 2215 H=2215:00 +CPU= 51.265
  04:07:12 STEP 2216 H=2216:00 +CPU= 57.897
  04:08:37 STEP 2217 H=2217:00 +CPU= 84.776
  04:09:32 STEP 2218 H=2218:00 +CPU= 54.468
  04:10:27 STEP 2219 H=2219:00 +CPU= 55.183
  04:12:02 STEP 2220 H=2220:00 +CPU= 94.379
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002220
  04:12:56 STEP 2221 H=2221:00 +CPU= 53.262
  04:13:55 STEP 2222 H=2222:00 +CPU= 58.213
  04:15:19 STEP 2223 H=2223:00 +CPU= 83.895
  04:16:17 STEP 2224 H=2224:00 +CPU= 57.005
  04:17:13 STEP 2225 H=2225:00 +CPU= 55.565
  04:18:37 STEP 2226 H=2226:00 +CPU= 83.661
  04:19:28 STEP 2227 H=2227:00 +CPU= 50.171
  04:20:20 STEP 2228 H=2228:00 +CPU= 51.335
  04:21:39 STEP 2229 H=2229:00 +CPU= 78.277
  04:22:30 STEP 2230 H=2230:00 +CPU= 51.422
  04:23:22 STEP 2231 H=2231:00 +CPU= 51.726
  04:24:54 STEP 2232 H=2232:00 +CPU= 90.649
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMGGhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMSHhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165268/ICMUAhpi1+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2179_2021050100_123_947_12165268_0_r720503287_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 8035200
  04:25:43 STEP 2233 H=2233:00 +CPU= 49.327
  04:26:35 STEP 2234 H=2234:00 +CPU= 51.292
  04:27:53 STEP 2235 H=2235:00 +CPU= 77.357
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:56:47 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 8,035,200 1,644 0.0002
09 May 2023 15:56:47 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,948,800 3,322 0.0004
09 May 2023 15:56:42 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,862,400 1,716 0.0002
09 May 2023 15:56:42 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,776,000 3,320 0.0004
09 May 2023 15:56:36 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,689,600 1,713 0.0002
09 May 2023 15:56:36 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,603,200 3,291 0.0004
09 May 2023 15:56:31 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,516,800 1,695 0.0002
09 May 2023 15:56:31 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,430,400 3,266 0.0004
09 May 2023 15:56:26 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,344,000 1,664 0.0002
09 May 2023 15:56:26 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,257,600 3,291 0.0005
09 May 2023 15:56:21 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,171,200 1,693 0.0002
09 May 2023 15:56:21 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 7,084,800 3,230 0.0005
09 May 2023 15:56:16 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,998,400 1,644 0.0002
09 May 2023 15:56:16 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,912,000 3,279 0.0005
09 May 2023 15:56:11 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,825,600 1,694 0.0002
09 May 2023 15:56:11 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,739,200 3,280 0.0005
09 May 2023 15:56:06 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,652,800 1,685 0.0003
09 May 2023 15:56:06 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,566,400 3,279 0.0005
09 May 2023 15:56:01 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,480,000 1,693 0.0003
09 May 2023 15:56:01 1520117 22247284 oifs_43r3_ps_2179_2021050100_123_947_12165268_0 6,393,600 3,247 0.0005


©2024 climateprediction.net