climateprediction.net home page
Task 22246172

Task 22246172

Name oifs_43r3_ps_1090_2021050100_123_946_12164179_0
Workunit 12164179
Created 29 Nov 2022, 10:32:13 UTC
Sent 29 Nov 2022, 10:36:39 UTC
Report deadline 29 Dec 2022, 10:36:39 UTC
Received 26 Dec 2022, 10:27:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1520117
Run time 1 days 23 hours 53 min 37 sec
CPU time 28 min 47 sec
Validate state Invalid
Credit 11,612.16
Device peak FLOPS 3.37 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,419.07 MB
Peak swap size 4,978.16 MB
Peak disk usage 38,490.48 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ignal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:61995:61995] [20221225:234207:1672033327:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:61995:61995] [20221225:234207:1672033327:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:43:51 STEP 2088 H=2088:00 +CPU= 97.784
  23:44:43 STEP 2089 H=2089:00 +CPU= 51.734
  23:45:36 STEP 2090 H=2090:00 +CPU= 52.147
  23:47:01 STEP 2091 H=2091:00 +CPU= 85.074
  23:47:59 STEP 2092 H=2092:00 +CPU= 56.814
  23:48:50 STEP 2093 H=2093:00 +CPU= 50.801
  23:50:15 STEP 2094 H=2094:00 +CPU= 85.020
  23:51:08 STEP 2095 H=2095:00 +CPU= 51.824
  23:52:04 STEP 2096 H=2096:00 +CPU= 55.763
  23:53:29 STEP 2097 H=2097:00 +CPU= 85.155
  23:54:25 STEP 2098 H=2098:00 +CPU= 55.558
  23:55:17 STEP 2099 H=2099:00 +CPU= 51.259
  23:56:51 STEP 2100 H=2100:00 +CPU= 93.020
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002100
  23:57:43 STEP 2101 H=2101:00 +CPU= 52.122
  23:58:36 STEP 2102 H=2102:00 +CPU= 52.689
  23:59:59 STEP 2103 H=2103:00 +CPU= 82.510
  00:00:55 STEP 2104 H=2104:00 +CPU= 55.207
  00:01:47 STEP 2105 H=2105:00 +CPU= 52.130
  00:03:13 STEP 2106 H=2106:00 +CPU= 85.007
  00:04:05 STEP 2107 H=2107:00 +CPU= 51.666
  00:04:58 STEP 2108 H=2108:00 +CPU= 52.259
  00:06:22 STEP 2109 H=2109:00 +CPU= 83.305
  00:07:16 STEP 2110 H=2110:00 +CPU= 53.428
  00:08:10 STEP 2111 H=2111:00 +CPU= 53.787
  00:09:47 STEP 2112 H=2112:00 +CPU= 96.470
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7603200
  00:10:41 STEP 2113 H=2113:00 +CPU= 52.988
  00:11:38 STEP 2114 H=2114:00 +CPU= 56.558
  00:13:04 STEP 2115 H=2115:00 +CPU= 85.221
  00:13:59 STEP 2116 H=2116:00 +CPU= 55.027
  00:14:52 STEP 2117 H=2117:00 +CPU= 51.985
  00:16:20 STEP 2118 H=2118:00 +CPU= 87.540
  00:17:14 STEP 2119 H=2119:00 +CPU= 53.052
  00:18:10 STEP 2120 H=2120:00 +CPU= 55.777
  00:19:36 STEP 2121 H=2121:00 +CPU= 85.350
  00:20:33 STEP 2122 H=2122:00 +CPU= 56.410
  00:21:28 STEP 2123 H=2123:00 +CPU= 54.969
  00:23:05 STEP 2124 H=2124:00 +CPU= 95.857
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002124
  00:24:00 STEP 2125 H=2125:00 +CPU= 55.299
  00:24:58 STEP 2126 H=2126:00 +CPU= 57.644
  00:26:24 STEP 2127 H=2127:00 +CPU= 85.144
  00:27:20 STEP 2128 H=2128:00 +CPU= 55.841
  00:28:16 STEP 2129 H=2129:00 +CPU= 55.672
  00:29:42 STEP 2130 H=2130:00 +CPU= 85.202
  00:30:34 STEP 2131 H=2131:00 +CPU= 50.908
  00:31:29 STEP 2132 H=2132:00 +CPU= 55.589
  00:32:56 STEP 2133 H=2133:00 +CPU= 82.076
  00:33:52 STEP 2134 H=2134:00 +CPU= 56.026
  00:34:48 STEP 2135 H=2135:00 +CPU= 55.168
  00:36:28 STEP 2136 H=2136:00 +CPU= 99.544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 7689600
  00:37:22 STEP 2137 H=2137:00 +CPU= 54.118
  00:38:19 STEP 2138 H=2138:00 +CPU= 56.109
  00:39:46 STEP 2139 H=2139:00 +CPU= 86.556
  00:40:42 STEP 2140 H=2140:00 +CPU= 55.917
  00:41:34 STEP 2141 H=2141:00 +CPU= 51.216
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1090
(argv4) batchid: 946
(argv5) wuid: 12164179
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2172590d6cefc0a114942eb0f84c8da to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
ic_ancil_file: ic_ancil_12164179
ifsdata_file: ifsdata_12164179
climate_data_file: clim_data_12164179
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_b0d3c690ccc453bb4dc076441c4d9637 to: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34b4245e3baed7acd76267410d8bf0e1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451
The child process has been launched with process id: 63127
Executing the command: /var/lib/boinc-client/slots/2/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63127:63127] [20221226:004328:1672037008:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:45:15 STEP 2136 H=2136:00 +CPU=101.448
  00:46:09 STEP 2137 H=2137:00 +CPU= 53.573
  00:47:02 STEP 2138 H=2138:00 +CPU= 52.961
  00:48:28 STEP 2139 H=2139:00 +CPU= 85.930
  00:49:23 STEP 2140 H=2140:00 +CPU= 54.176
  00:50:18 STEP 2141 H=2141:00 +CPU= 54.869
  00:51:47 STEP 2142 H=2142:00 +CPU= 88.359
  00:52:40 STEP 2143 H=2143:00 +CPU= 52.852
  00:53:35 STEP 2144 H=2144:00 +CPU= 54.315
  00:55:02 STEP 2145 H=2145:00 +CPU= 86.885
  00:55:59 STEP 2146 H=2146:00 +CPU= 56.533
  00:56:50 STEP 2147 H=2147:00 +CPU= 50.696
  00:58:23 STEP 2148 H=2148:00 +CPU= 92.605
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002148
  00:59:17 STEP 2149 H=2149:00 +CPU= 52.964
  01:00:10 STEP 2150 H=2150:00 +CPU= 52.953
  01:01:35 STEP 2151 H=2151:00 +CPU= 84.009
  01:02:30 STEP 2152 H=2152:00 +CPU= 55.099
  01:03:26 STEP 2153 H=2153:00 +CPU= 55.561
  01:04:55 STEP 2154 H=2154:00 +CPU= 88.549
  01:05:50 STEP 2155 H=2155:00 +CPU= 54.670
  01:06:48 STEP 2156 H=2156:00 +CPU= 57.272
  01:08:14 STEP 2157 H=2157:00 +CPU= 86.082
  01:09:11 STEP 2158 H=2158:00 +CPU= 55.928
  01:10:07 STEP 2159 H=2159:00 +CPU= 56.071
  01:11:47 STEP 2160 H=2160:00 +CPU= 99.303
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7776000
  01:12:42 STEP 2161 H=2161:00 +CPU= 54.441
  01:13:38 STEP 2162 H=2162:00 +CPU= 55.371
  01:15:04 STEP 2163 H=2163:00 +CPU= 85.354
  01:16:01 STEP 2164 H=2164:00 +CPU= 56.214
  01:16:55 STEP 2165 H=2165:00 +CPU= 54.598
  01:18:24 STEP 2166 H=2166:00 +CPU= 88.040
  01:19:19 STEP 2167 H=2167:00 +CPU= 53.935
  01:20:16 STEP 2168 H=2168:00 +CPU= 56.631
  01:21:42 STEP 2169 H=2169:00 +CPU= 85.886
  01:22:39 STEP 2170 H=2170:00 +CPU= 56.227
  01:23:32 STEP 2171 H=2171:00 +CPU= 52.922
  01:25:07 STEP 2172 H=2172:00 +CPU= 94.714
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002172
  01:26:02 STEP 2173 H=2173:00 +CPU= 54.054
  01:27:00 STEP 2174 H=2174:00 +CPU= 57.798
  01:28:27 STEP 2175 H=2175:00 +CPU= 86.073
  01:29:25 STEP 2176 H=2176:00 +CPU= 57.455
  01:30:22 STEP 2177 H=2177:00 +CPU= 56.971
  01:31:51 STEP 2178 H=2178:00 +CPU= 87.869
  01:32:47 STEP 2179 H=2179:00 +CPU= 55.707
  01:33:44 STEP 2180 H=2180:00 +CPU= 56.587
  01:35:08 STEP 2181 H=2181:00 +CPU= 84.263
  01:36:05 STEP 2182 H=2182:00 +CPU= 56.040
  01:37:00 STEP 2183 H=2183:00 +CPU= 54.640
  01:38:40 STEP 2184 H=2184:00 +CPU= 99.626
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 7862400
  01:39:35 STEP 2185 H=2185:00 +CPU= 54.443
  01:40:29 STEP 2186 H=2186:00 +CPU= 53.775
  01:41:56 STEP 2187 H=2187:00 +CPU= 85.377
  01:42:53 STEP 2188 H=2188:00 +CPU= 56.910
  01:43:46 STEP 2189 H=2189:00 +CPU= 53.235
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1090
(argv4) batchid: 946
(argv5) wuid: 12164179
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2172590d6cefc0a114942eb0f84c8da to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
ic_ancil_file: ic_ancil_12164179
ifsdata_file: ifsdata_12164179
climate_data_file: clim_data_12164179
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_b0d3c690ccc453bb4dc076441c4d9637 to: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34b4245e3baed7acd76267410d8bf0e1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451
The child process has been launched with process id: 63666
Executing the command: /var/lib/boinc-client/slots/2/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:63666:63666] [20221226:014518:1672040718:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:47:02 STEP 2184 H=2184:00 +CPU= 98.815
  01:47:58 STEP 2185 H=2185:00 +CPU= 55.088
  01:48:53 STEP 2186 H=2186:00 +CPU= 54.597
  01:50:22 STEP 2187 H=2187:00 +CPU= 88.775
  01:51:18 STEP 2188 H=2188:00 +CPU= 55.628
  01:52:12 STEP 2189 H=2189:00 +CPU= 54.443
  01:53:38 STEP 2190 H=2190:00 +CPU= 84.701
  01:54:33 STEP 2191 H=2191:00 +CPU= 55.402
  01:55:29 STEP 2192 H=2192:00 +CPU= 55.243
  01:56:57 STEP 2193 H=2193:00 +CPU= 87.194
  01:57:52 STEP 2194 H=2194:00 +CPU= 54.947
  01:58:46 STEP 2195 H=2195:00 +CPU= 53.534
  02:00:23 STEP 2196 H=2196:00 +CPU= 96.440
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002196
  02:01:16 STEP 2197 H=2197:00 +CPU= 52.858
  02:02:11 STEP 2198 H=2198:00 +CPU= 55.550
  02:03:38 STEP 2199 H=2199:00 +CPU= 86.273
  02:04:32 STEP 2200 H=2200:00 +CPU= 53.434
  02:05:27 STEP 2201 H=2201:00 +CPU= 54.223
  02:06:56 STEP 2202 H=2202:00 +CPU= 89.445
  02:07:50 STEP 2203 H=2203:00 +CPU= 53.591
  02:08:47 STEP 2204 H=2204:00 +CPU= 56.432
  02:10:13 STEP 2205 H=2205:00 +CPU= 85.740
  02:11:08 STEP 2206 H=2206:00 +CPU= 54.403
  02:12:06 STEP 2207 H=2207:00 +CPU= 57.743
  02:13:48 STEP 2208 H=2208:00 +CPU=100.841
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 7948800
  02:14:41 STEP 2209 H=2209:00 +CPU= 52.070
  02:15:36 STEP 2210 H=2210:00 +CPU= 55.424
  02:17:04 STEP 2211 H=2211:00 +CPU= 87.363
  02:17:59 STEP 2212 H=2212:00 +CPU= 54.671
  02:18:55 STEP 2213 H=2213:00 +CPU= 55.063
  02:20:26 STEP 2214 H=2214:00 +CPU= 90.364
  02:21:21 STEP 2215 H=2215:00 +CPU= 54.552
  02:22:17 STEP 2216 H=2216:00 +CPU= 56.103
  02:23:46 STEP 2217 H=2217:00 +CPU= 87.890
  02:24:40 STEP 2218 H=2218:00 +CPU= 54.068
  02:25:35 STEP 2219 H=2219:00 +CPU= 54.352
  02:27:12 STEP 2220 H=2220:00 +CPU= 96.809
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002220
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002220
  02:28:05 STEP 2221 H=2221:00 +CPU= 52.146
  02:29:02 STEP 2222 H=2222:00 +CPU= 56.776
  02:30:30 STEP 2223 H=2223:00 +CPU= 87.314
  02:31:26 STEP 2224 H=2224:00 +CPU= 56.345
  02:32:20 STEP 2225 H=2225:00 +CPU= 53.537
  02:33:50 STEP 2226 H=2226:00 +CPU= 89.730
  02:34:44 STEP 2227 H=2227:00 +CPU= 52.701
  02:35:41 STEP 2228 H=2228:00 +CPU= 56.713
  02:37:10 STEP 2229 H=2229:00 +CPU= 87.887
  02:38:05 STEP 2230 H=2230:00 +CPU= 54.692
  02:39:01 STEP 2231 H=2231:00 +CPU= 55.927
  02:40:45 STEP 2232 H=2232:00 +CPU=102.976
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 8035200
  02:41:40 STEP 2233 H=2233:00 +CPU= 54.695
  02:42:36 STEP 2234 H=2234:00 +CPU= 55.987
  02:44:04 STEP 2235 H=2235:00 +CPU= 87.380
  02:44:58 STEP 2236 H=2236:00 +CPU= 53.476
  02:45:55 STEP 2237 H=2237:00 +CPU= 56.413
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1090
(argv4) batchid: 946
(argv5) wuid: 12164179
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2172590d6cefc0a114942eb0f84c8da to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
ic_ancil_file: ic_ancil_12164179
ifsdata_file: ifsdata_12164179
climate_data_file: clim_data_12164179
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_b0d3c690ccc453bb4dc076441c4d9637 to: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34b4245e3baed7acd76267410d8bf0e1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451
The child process has been launched with process id: 64227
Executing the command: /var/lib/boinc-client/slots/2/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64227:64227] [20221226:024752:1672044472:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:49:35 STEP 2232 H=2232:00 +CPU= 97.031
  02:50:27 STEP 2233 H=2233:00 +CPU= 52.020
  02:51:24 STEP 2234 H=2234:00 +CPU= 56.653
  02:52:48 STEP 2235 H=2235:00 +CPU= 82.961
  02:53:41 STEP 2236 H=2236:00 +CPU= 52.672
  02:54:33 STEP 2237 H=2237:00 +CPU= 52.303
  02:55:59 STEP 2238 H=2238:00 +CPU= 85.038
  02:56:53 STEP 2239 H=2239:00 +CPU= 53.404
  02:57:50 STEP 2240 H=2240:00 +CPU= 56.587
  02:59:16 STEP 2241 H=2241:00 +CPU= 86.013
  03:00:12 STEP 2242 H=2242:00 +CPU= 56.001
  03:01:07 STEP 2243 H=2243:00 +CPU= 54.288
  03:02:43 STEP 2244 H=2244:00 +CPU= 95.809
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002244
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002244
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002244
  03:03:38 STEP 2245 H=2245:00 +CPU= 54.100
  03:04:34 STEP 2246 H=2246:00 +CPU= 56.455
  03:06:01 STEP 2247 H=2247:00 +CPU= 86.379
  03:06:57 STEP 2248 H=2248:00 +CPU= 55.043
  03:07:51 STEP 2249 H=2249:00 +CPU= 54.349
  03:09:21 STEP 2250 H=2250:00 +CPU= 87.889
  03:10:13 STEP 2251 H=2251:00 +CPU= 52.611
  03:11:09 STEP 2252 H=2252:00 +CPU= 54.503
  03:12:35 STEP 2253 H=2253:00 +CPU= 85.396
  03:13:32 STEP 2254 H=2254:00 +CPU= 57.136
  03:14:28 STEP 2255 H=2255:00 +CPU= 55.533
  03:16:09 STEP 2256 H=2256:00 +CPU= 99.977
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002256
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002256
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 8121600
  03:17:05 STEP 2257 H=2257:00 +CPU= 55.699
  03:18:02 STEP 2258 H=2258:00 +CPU= 56.537
  03:19:28 STEP 2259 H=2259:00 +CPU= 85.643
  03:20:24 STEP 2260 H=2260:00 +CPU= 55.455
  03:21:19 STEP 2261 H=2261:00 +CPU= 54.343
  03:22:48 STEP 2262 H=2262:00 +CPU= 88.588
  03:23:41 STEP 2263 H=2263:00 +CPU= 53.497
  03:24:37 STEP 2264 H=2264:00 +CPU= 54.787
  03:26:04 STEP 2265 H=2265:00 +CPU= 86.679
  03:27:02 STEP 2266 H=2266:00 +CPU= 57.168
  03:27:55 STEP 2267 H=2267:00 +CPU= 52.806
  03:29:32 STEP 2268 H=2268:00 +CPU= 96.882
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002268
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002268
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002268
  03:30:23 STEP 2269 H=2269:00 +CPU= 50.009
  03:31:18 STEP 2270 H=2270:00 +CPU= 54.945
  03:32:45 STEP 2271 H=2271:00 +CPU= 86.441
  03:33:39 STEP 2272 H=2272:00 +CPU= 53.277
  03:34:33 STEP 2273 H=2273:00 +CPU= 53.642
  03:36:02 STEP 2274 H=2274:00 +CPU= 88.460
  03:36:57 STEP 2275 H=2275:00 +CPU= 54.908
  03:37:52 STEP 2276 H=2276:00 +CPU= 54.523
  03:39:19 STEP 2277 H=2277:00 +CPU= 86.932
  03:40:17 STEP 2278 H=2278:00 +CPU= 57.244
  03:41:12 STEP 2279 H=2279:00 +CPU= 55.069
  03:42:53 STEP 2280 H=2280:00 +CPU=100.123
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002280
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002280
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 8208000
  03:43:48 STEP 2281 H=2281:00 +CPU= 54.708
  03:44:45 STEP 2282 H=2282:00 +CPU= 56.079
  03:46:12 STEP 2283 H=2283:00 +CPU= 86.267
  03:47:08 STEP 2284 H=2284:00 +CPU= 55.749
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1090
(argv4) batchid: 946
(argv5) wuid: 12164179
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2172590d6cefc0a114942eb0f84c8da to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_1090_2021050100_123_946_12164179.zip
ic_ancil_file: ic_ancil_12164179
ifsdata_file: ifsdata_12164179
climate_data_file: clim_data_12164179
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_b0d3c690ccc453bb4dc076441c4d9637 to: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12164179.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34b4245e3baed7acd76267410d8bf0e1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12164179.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12164179.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451
The child process has been launched with process id: 64771
Executing the command: /var/lib/boinc-client/slots/2/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:hc2u20-coachalfonsosrestivo-soccerservices:1:1:64771:64771] [20221226:034859:1672048139:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:50:42 STEP 2280 H=2280:00 +CPU= 97.075
  03:51:36 STEP 2281 H=2281:00 +CPU= 53.742
  03:52:32 STEP 2282 H=2282:00 +CPU= 55.341
  03:54:00 STEP 2283 H=2283:00 +CPU= 87.244
  03:54:56 STEP 2284 H=2284:00 +CPU= 55.453
  03:55:49 STEP 2285 H=2285:00 +CPU= 52.876
  03:57:20 STEP 2286 H=2286:00 +CPU= 90.811
  03:58:14 STEP 2287 H=2287:00 +CPU= 53.738
  03:59:10 STEP 2288 H=2288:00 +CPU= 56.079
  04:00:37 STEP 2289 H=2289:00 +CPU= 85.727
  04:01:31 STEP 2290 H=2290:00 +CPU= 53.770
  04:02:23 STEP 2291 H=2291:00 +CPU= 51.393
  04:03:55 STEP 2292 H=2292:00 +CPU= 91.502
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002292
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002292
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002292
  04:04:47 STEP 2293 H=2293:00 +CPU= 51.412
  04:05:42 STEP 2294 H=2294:00 +CPU= 55.169
  04:07:09 STEP 2295 H=2295:00 +CPU= 85.767
  04:08:04 STEP 2296 H=2296:00 +CPU= 55.041
  04:08:59 STEP 2297 H=2297:00 +CPU= 54.260
  04:10:26 STEP 2298 H=2298:00 +CPU= 86.782
  04:11:19 STEP 2299 H=2299:00 +CPU= 52.990
  04:12:16 STEP 2300 H=2300:00 +CPU= 56.675
  04:13:41 STEP 2301 H=2301:00 +CPU= 84.539
  04:14:38 STEP 2302 H=2302:00 +CPU= 55.799
  04:15:35 STEP 2303 H=2303:00 +CPU= 57.041
  04:17:16 STEP 2304 H=2304:00 +CPU=100.378
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002304
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002304
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMGGhpi1+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMSHhpi1+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164179/ICMUAhpi1+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1090_2021050100_123_946_12164179_0_r2070515451_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 8294400
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:56:47 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 8,294,400 1,694 0.0002
09 May 2023 15:56:47 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 8,208,000 3,282 0.0004
09 May 2023 15:56:42 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 8,121,600 1,685 0.0002
09 May 2023 15:56:42 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 8,035,200 3,312 0.0004
09 May 2023 15:56:36 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,948,800 1,706 0.0002
09 May 2023 15:56:36 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,862,400 3,291 0.0004
09 May 2023 15:56:31 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,776,000 1,696 0.0002
09 May 2023 15:56:31 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,689,600 3,235 0.0004
09 May 2023 15:56:26 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,603,200 1,655 0.0002
09 May 2023 15:56:26 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,516,800 3,288 0.0004
09 May 2023 15:56:21 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,430,400 1,703 0.0002
09 May 2023 15:56:21 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,344,000 3,242 0.0004
09 May 2023 15:56:16 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,257,600 1,674 0.0002
09 May 2023 15:56:16 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,171,200 3,249 0.0005
09 May 2023 15:56:11 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 7,084,800 1,673 0.0002
09 May 2023 15:56:11 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 6,998,400 3,306 0.0005
09 May 2023 15:56:06 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 6,912,000 1,692 0.0002
09 May 2023 15:56:06 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 6,825,600 3,270 0.0005
09 May 2023 15:56:01 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 6,739,200 1,683 0.0002
09 May 2023 15:56:01 1520117 22246172 oifs_43r3_ps_1090_2021050100_123_946_12164179_0 6,652,800 3,250 0.0005


©2024 climateprediction.net