climateprediction.net home page
Task 22290117

Task 22290117

Name oifs_43r3_ps_0006_2018050100_123_987_12203650_0
Workunit 12203650
Created 24 Dec 2022, 19:30:31 UTC
Sent 16 Jan 2023, 18:46:57 UTC
Report deadline 17 Mar 2023, 18:46:57 UTC
Received 20 Jan 2023, 20:15:34 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532028
Run time 10 hours 42 min 24 sec
CPU time 10 hours 21 min 50 sec
Validate state Invalid
Credit 7,136.64
Device peak FLOPS 8.61 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,779.24 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.33 MB
Stderr 
<core_client_version>7.20.4</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_45.zip
Uploading the intermediate file: upload_file_45.zip
  19:38:52 STEP 1105 H=1105:00 +CPU= 18.508
  19:39:13 STEP 1106 H=1106:00 +CPU= 19.743
  19:39:43 STEP 1107 H=1107:00 +CPU= 29.625
  19:40:03 STEP 1108 H=1108:00 +CPU= 19.512
  19:40:23 STEP 1109 H=1109:00 +CPU= 19.635
  19:40:53 STEP 1110 H=1110:00 +CPU= 29.251
  19:41:12 STEP 1111 H=1111:00 +CPU= 19.029
  19:41:32 STEP 1112 H=1112:00 +CPU= 19.410
  19:42:02 STEP 1113 H=1113:00 +CPU= 28.729
  19:42:21 STEP 1114 H=1114:00 +CPU= 18.745
  19:42:39 STEP 1115 H=1115:00 +CPU= 18.062
  19:43:11 STEP 1116 H=1116:00 +CPU= 30.376
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001116
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001116
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001116
  19:43:29 STEP 1117 H=1117:00 +CPU= 18.367
  19:43:50 STEP 1118 H=1118:00 +CPU= 19.844
  19:44:20 STEP 1119 H=1119:00 +CPU= 29.770
  19:44:41 STEP 1120 H=1120:00 +CPU= 20.226
  19:45:01 STEP 1121 H=1121:00 +CPU= 19.715
  19:45:32 STEP 1122 H=1122:00 +CPU= 30.194
  19:45:50 STEP 1123 H=1123:00 +CPU= 17.777
  19:46:09 STEP 1124 H=1124:00 +CPU= 18.425
  19:46:35 STEP 1125 H=1125:00 +CPU= 24.942
  19:46:49 STEP 1126 H=1126:00 +CPU= 13.956
  19:47:02 STEP 1127 H=1127:00 +CPU= 13.412
  19:47:26 STEP 1128 H=1128:00 +CPU= 22.938
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001128
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001128
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001128
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001104
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001104
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001104
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001116
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001116
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001116
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_46.zip
Uploading the intermediate file: upload_file_46.zip
  19:47:38 STEP 1129 H=1129:00 +CPU= 12.493
  19:47:51 STEP 1130 H=1130:00 +CPU= 12.347
  19:48:10 STEP 1131 H=1131:00 +CPU= 19.156
  19:48:27 STEP 1132 H=1132:00 +CPU= 15.879
  19:48:46 STEP 1133 H=1133:00 +CPU= 18.721
  19:49:15 STEP 1134 H=1134:00 +CPU= 28.597
  19:49:35 STEP 1135 H=1135:00 +CPU= 19.632
  19:49:55 STEP 1136 H=1136:00 +CPU= 19.762
  19:50:26 STEP 1137 H=1137:00 +CPU= 30.768
  19:50:48 STEP 1138 H=1138:00 +CPU= 21.492
  19:51:07 STEP 1139 H=1139:00 +CPU= 19.128
  19:51:39 STEP 1140 H=1140:00 +CPU= 31.935
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001140
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001140
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001140
  19:51:54 STEP 1141 H=1141:00 +CPU= 14.799
  19:52:08 STEP 1142 H=1142:00 +CPU= 13.977
  19:52:34 STEP 1143 H=1143:00 +CPU= 25.609
  19:52:54 STEP 1144 H=1144:00 +CPU= 20.462
  19:53:15 STEP 1145 H=1145:00 +CPU= 20.324
  19:53:46 STEP 1146 H=1146:00 +CPU= 31.233
  19:54:07 STEP 1147 H=1147:00 +CPU= 20.530
  19:54:28 STEP 1148 H=1148:00 +CPU= 20.471
  19:55:00 STEP 1149 H=1149:00 +CPU= 32.305
  19:55:23 STEP 1150 H=1150:00 +CPU= 21.390
  19:55:40 STEP 1151 H=1151:00 +CPU= 16.619
  19:56:10 STEP 1152 H=1152:00 +CPU= 28.510
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001152
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001152
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001152
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001128
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001128
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001128
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001140
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001140
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001140
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_47.zip
Uploading the intermediate file: upload_file_47.zip
  19:56:29 STEP 1153 H=1153:00 +CPU= 18.361
  19:56:47 STEP 1154 H=1154:00 +CPU= 17.391
  19:57:15 STEP 1155 H=1155:00 +CPU= 27.840
  19:57:30 STEP 1156 H=1156:00 +CPU= 14.554
  19:57:43 STEP 1157 H=1157:00 +CPU= 13.085
  19:58:06 STEP 1158 H=1158:00 +CPU= 22.112
  19:58:25 STEP 1159 H=1159:00 +CPU= 18.172
  19:58:44 STEP 1160 H=1160:00 +CPU= 18.425
  19:59:14 STEP 1161 H=1161:00 +CPU= 29.376
  19:59:35 STEP 1162 H=1162:00 +CPU= 21.085
  19:59:56 STEP 1163 H=1163:00 +CPU= 20.711
  20:00:32 STEP 1164 H=1164:00 +CPU= 34.637
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001164
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001164
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001164
  20:00:54 STEP 1165 H=1165:00 +CPU= 21.082
  20:01:15 STEP 1166 H=1166:00 +CPU= 20.919
  20:01:47 STEP 1167 H=1167:00 +CPU= 31.353
  20:02:09 STEP 1168 H=1168:00 +CPU= 20.913
  20:02:32 STEP 1169 H=1169:00 +CPU= 21.550
  20:03:04 STEP 1170 H=1170:00 +CPU= 32.003
  20:03:27 STEP 1171 H=1171:00 +CPU= 21.297
  20:03:48 STEP 1172 H=1172:00 +CPU= 21.329
  20:04:20 STEP 1173 H=1173:00 +CPU= 31.224
  20:04:43 STEP 1174 H=1174:00 +CPU= 22.147
  20:05:04 STEP 1175 H=1175:00 +CPU= 20.471
  20:05:41 STEP 1176 H=1176:00 +CPU= 35.441
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001176
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001176
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001176
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001152
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001152
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001152
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001164
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001164
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001164
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_48.zip
Uploading the intermediate file: upload_file_48.zip
  20:06:01 STEP 1177 H=1177:00 +CPU= 20.142
  20:06:23 STEP 1178 H=1178:00 +CPU= 20.757
  20:06:55 STEP 1179 H=1179:00 +CPU= 31.088
  20:07:16 STEP 1180 H=1180:00 +CPU= 20.351
  20:07:35 STEP 1181 H=1181:00 +CPU= 19.217
  20:08:07 STEP 1182 H=1182:00 +CPU= 31.340
  20:08:28 STEP 1183 H=1183:00 +CPU= 20.751
  20:08:49 STEP 1184 H=1184:00 +CPU= 20.717
  20:09:20 STEP 1185 H=1185:00 +CPU= 31.405
  20:09:41 STEP 1186 H=1186:00 +CPU= 20.486
  20:10:02 STEP 1187 H=1187:00 +CPU= 20.509
  20:10:38 STEP 1188 H=1188:00 +CPU= 35.399
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001188
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001188
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001188
  20:10:59 STEP 1189 H=1189:00 +CPU= 21.058
  20:11:21 STEP 1190 H=1190:00 +CPU= 21.454
  20:11:54 STEP 1191 H=1191:00 +CPU= 32.281
  20:12:17 STEP 1192 H=1192:00 +CPU= 22.026
  20:12:38 STEP 1193 H=1193:00 +CPU= 20.699
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0006
(argv4) batchid: 987
(argv5) wuid: 12203650
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/home/boinc/slots/10
Project directory is: /mnt/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650
..mkdir for temp folder for results failed
Copying: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/home/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/home/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d4cc118baf285b3edee01c230448a2d3 to: /mnt/home/boinc/slots/10/oifs_43r3_ps_0006_2018050100_123_987_12203650.zip
Unzipping the namelist zip file: /mnt/home/boinc/slots/10/oifs_43r3_ps_0006_2018050100_123_987_12203650.zip
ic_ancil_file: ic_ancil_12203650
ifsdata_file: ifsdata_12203650
climate_data_file: clim_data_12203650
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2eb215ff8b4befb44926d7fc2f1a9295 to: /mnt/home/boinc/slots/10/ic_ancil_12203650.zip
Unzipping the IC ancils zip file: /mnt/home/boinc/slots/10/ic_ancil_12203650.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e5de5e7c9c0ffd484b6cd17c66e9be41 to: /mnt/home/boinc/slots/10/ifsdata/ifsdata_12203650.zip
Unzipping the ifsdata_zip file: /mnt/home/boinc/slots/10/ifsdata/ifsdata_12203650.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/home/boinc/slots/10/159l_2/clim_data_12203650.zip
Unzipping the climate data zip file: /mnt/home/boinc/slots/10/159l_2/clim_data_12203650.zip
Checking for progress XML file: /mnt/home/boinc/slots/10/progress_file_12203650.xml
Opened progress file ok : /mnt/home/boinc/slots/10/progress_file_12203650.xml
-- Model is restarting --
Adjusting last_iter, 1194, to previous model restart step.
Creating progress file: /mnt/home/boinc/slots/10/progress_file_12203650.xml
last_cpu_time: 30199
upload_file_number: 49
last_iter: 1175
last_upload: 4233600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973
The child process has been launched with process id: 24692
Executing the command: /mnt/home/boinc/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:24692:24692] [20230119:205416:1674158056:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:54:54 STEP 1176 H=1176:00 +CPU= 34.848
  20:55:14 STEP 1177 H=1177:00 +CPU= 19.656
  20:55:35 STEP 1178 H=1178:00 +CPU= 20.795
  20:56:06 STEP 1179 H=1179:00 +CPU= 30.396
  20:56:26 STEP 1180 H=1180:00 +CPU= 19.705
  20:56:45 STEP 1181 H=1181:00 +CPU= 19.109
  20:57:15 STEP 1182 H=1182:00 +CPU= 30.064
  20:57:35 STEP 1183 H=1183:00 +CPU= 19.562
  20:57:55 STEP 1184 H=1184:00 +CPU= 19.345
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0006
(argv4) batchid: 987
(argv5) wuid: 12203650
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/home/boinc/slots/10
Project directory is: /mnt/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650
..mkdir for temp folder for results failed
Copying: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/home/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/home/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d4cc118baf285b3edee01c230448a2d3 to: /mnt/home/boinc/slots/10/oifs_43r3_ps_0006_2018050100_123_987_12203650.zip
Unzipping the namelist zip file: /mnt/home/boinc/slots/10/oifs_43r3_ps_0006_2018050100_123_987_12203650.zip
ic_ancil_file: ic_ancil_12203650
ifsdata_file: ifsdata_12203650
climate_data_file: clim_data_12203650
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2eb215ff8b4befb44926d7fc2f1a9295 to: /mnt/home/boinc/slots/10/ic_ancil_12203650.zip
Unzipping the IC ancils zip file: /mnt/home/boinc/slots/10/ic_ancil_12203650.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e5de5e7c9c0ffd484b6cd17c66e9be41 to: /mnt/home/boinc/slots/10/ifsdata/ifsdata_12203650.zip
Unzipping the ifsdata_zip file: /mnt/home/boinc/slots/10/ifsdata/ifsdata_12203650.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/home/boinc/slots/10/159l_2/clim_data_12203650.zip
Unzipping the climate data zip file: /mnt/home/boinc/slots/10/159l_2/clim_data_12203650.zip
Checking for progress XML file: /mnt/home/boinc/slots/10/progress_file_12203650.xml
Opened progress file ok : /mnt/home/boinc/slots/10/progress_file_12203650.xml
-- Model is restarting --
Adjusting last_iter, 1185, to previous model restart step.
Creating progress file: /mnt/home/boinc/slots/10/progress_file_12203650.xml
last_cpu_time: 30425
upload_file_number: 49
last_iter: 1175
last_upload: 4233600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973
The child process has been launched with process id: 16835
Executing the command: /mnt/home/boinc/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:16835:16835] [20230120:181411:1674234851:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:14:54 STEP 1176 H=1176:00 +CPU= 33.892
  18:15:15 STEP 1177 H=1177:00 +CPU= 20.325
  18:15:37 STEP 1178 H=1178:00 +CPU= 21.169
  18:16:10 STEP 1179 H=1179:00 +CPU= 30.881
  18:16:32 STEP 1180 H=1180:00 +CPU= 20.802
  18:16:53 STEP 1181 H=1181:00 +CPU= 20.444
  18:17:25 STEP 1182 H=1182:00 +CPU= 30.920
  18:17:46 STEP 1183 H=1183:00 +CPU= 20.738
  18:18:08 STEP 1184 H=1184:00 +CPU= 21.653
  18:18:41 STEP 1185 H=1185:00 +CPU= 31.480
  18:19:03 STEP 1186 H=1186:00 +CPU= 21.882
  18:19:25 STEP 1187 H=1187:00 +CPU= 21.326
  18:20:01 STEP 1188 H=1188:00 +CPU= 34.271
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001188
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001188
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001188
  18:20:22 STEP 1189 H=1189:00 +CPU= 21.300
  18:20:44 STEP 1190 H=1190:00 +CPU= 21.071
  18:21:16 STEP 1191 H=1191:00 +CPU= 31.355
  18:21:38 STEP 1192 H=1192:00 +CPU= 21.234
  18:21:59 STEP 1193 H=1193:00 +CPU= 21.020
  18:22:32 STEP 1194 H=1194:00 +CPU= 32.165
  18:22:53 STEP 1195 H=1195:00 +CPU= 20.941
  18:23:15 STEP 1196 H=1196:00 +CPU= 21.525
  18:23:47 STEP 1197 H=1197:00 +CPU= 30.595
  18:24:08 STEP 1198 H=1198:00 +CPU= 20.710
  18:24:29 STEP 1199 H=1199:00 +CPU= 20.830
  18:25:08 STEP 1200 H=1200:00 +CPU= 35.922
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001200
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001200
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001200
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001176
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001176
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001176
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001188
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001188
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001188
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_49.zip
Uploading the intermediate file: upload_file_49.zip
  18:25:28 STEP 1201 H=1201:00 +CPU= 18.519
  18:25:47 STEP 1202 H=1202:00 +CPU= 18.854
  18:26:15 STEP 1203 H=1203:00 +CPU= 27.488
  18:26:34 STEP 1204 H=1204:00 +CPU= 19.058
  18:26:55 STEP 1205 H=1205:00 +CPU= 19.553
  18:27:26 STEP 1206 H=1206:00 +CPU= 29.720
  18:27:46 STEP 1207 H=1207:00 +CPU= 19.260
  18:28:06 STEP 1208 H=1208:00 +CPU= 19.626
  18:28:36 STEP 1209 H=1209:00 +CPU= 29.277
  18:28:59 STEP 1210 H=1210:00 +CPU= 21.399
  18:29:19 STEP 1211 H=1211:00 +CPU= 19.743
  18:29:53 STEP 1212 H=1212:00 +CPU= 32.794
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001212
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001212
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001212
  18:30:13 STEP 1213 H=1213:00 +CPU= 19.808
  18:30:34 STEP 1214 H=1214:00 +CPU= 20.181
  18:31:05 STEP 1215 H=1215:00 +CPU= 30.215
  18:31:26 STEP 1216 H=1216:00 +CPU= 20.481
  18:31:48 STEP 1217 H=1217:00 +CPU= 20.904
  18:32:19 STEP 1218 H=1218:00 +CPU= 30.077
  18:32:40 STEP 1219 H=1219:00 +CPU= 19.760
  18:33:01 STEP 1220 H=1220:00 +CPU= 19.953
  18:33:31 STEP 1221 H=1221:00 +CPU= 29.661
  18:33:52 STEP 1222 H=1222:00 +CPU= 19.863
  18:34:13 STEP 1223 H=1223:00 +CPU= 20.492
  18:34:48 STEP 1224 H=1224:00 +CPU= 34.348
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001224
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001224
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001224
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001200
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001200
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001200
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001212
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001212
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001212
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_50.zip
Uploading the intermediate file: upload_file_50.zip
  18:35:09 STEP 1225 H=1225:00 +CPU= 19.639
  18:35:30 STEP 1226 H=1226:00 +CPU= 20.647
  18:36:01 STEP 1227 H=1227:00 +CPU= 30.381
  18:36:22 STEP 1228 H=1228:00 +CPU= 20.657
  18:36:43 STEP 1229 H=1229:00 +CPU= 19.805
  18:37:15 STEP 1230 H=1230:00 +CPU= 30.970
  18:37:35 STEP 1231 H=1231:00 +CPU= 19.899
  18:37:56 STEP 1232 H=1232:00 +CPU= 20.298
  18:38:27 STEP 1233 H=1233:00 +CPU= 30.802
  18:38:49 STEP 1234 H=1234:00 +CPU= 20.548
  18:39:10 STEP 1235 H=1235:00 +CPU= 20.238
  18:39:45 STEP 1236 H=1236:00 +CPU= 33.923
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001236
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001236
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001236
  18:40:06 STEP 1237 H=1237:00 +CPU= 20.556
  18:40:29 STEP 1238 H=1238:00 +CPU= 21.720
  18:41:01 STEP 1239 H=1239:00 +CPU= 30.566
  18:41:22 STEP 1240 H=1240:00 +CPU= 20.055
  18:41:42 STEP 1241 H=1241:00 +CPU= 19.562
  18:42:15 STEP 1242 H=1242:00 +CPU= 30.834
  18:42:36 STEP 1243 H=1243:00 +CPU= 19.942
  18:42:57 STEP 1244 H=1244:00 +CPU= 20.064
  18:43:28 STEP 1245 H=1245:00 +CPU= 30.132
  18:43:49 STEP 1246 H=1246:00 +CPU= 20.145
  18:44:10 STEP 1247 H=1247:00 +CPU= 20.580
  18:44:48 STEP 1248 H=1248:00 +CPU= 36.452
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001248
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001248
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001248
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001224
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001224
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001224
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001236
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001236
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001236
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_51.zip
Uploading the intermediate file: upload_file_51.zip
  18:45:08 STEP 1249 H=1249:00 +CPU= 20.085
  18:45:29 STEP 1250 H=1250:00 +CPU= 20.306
  18:46:01 STEP 1251 H=1251:00 +CPU= 30.441
  18:46:22 STEP 1252 H=1252:00 +CPU= 20.261
  18:46:42 STEP 1253 H=1253:00 +CPU= 20.017
  18:47:13 STEP 1254 H=1254:00 +CPU= 30.597
  18:47:34 STEP 1255 H=1255:00 +CPU= 19.944
  18:47:54 STEP 1256 H=1256:00 +CPU= 19.952
  18:48:26 STEP 1257 H=1257:00 +CPU= 30.324
  18:48:47 STEP 1258 H=1258:00 +CPU= 20.626
  18:49:09 STEP 1259 H=1259:00 +CPU= 20.553
  18:49:43 STEP 1260 H=1260:00 +CPU= 32.833
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001260
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001260
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001260
  18:50:04 STEP 1261 H=1261:00 +CPU= 19.889
  18:50:25 STEP 1262 H=1262:00 +CPU= 20.089
  18:50:56 STEP 1263 H=1263:00 +CPU= 29.712
  18:51:17 STEP 1264 H=1264:00 +CPU= 19.845
  18:51:36 STEP 1265 H=1265:00 +CPU= 19.177
  18:52:07 STEP 1266 H=1266:00 +CPU= 29.834
  18:52:28 STEP 1267 H=1267:00 +CPU= 19.877
  18:52:50 STEP 1268 H=1268:00 +CPU= 20.784
  18:53:21 STEP 1269 H=1269:00 +CPU= 30.495
  18:53:43 STEP 1270 H=1270:00 +CPU= 20.635
  18:54:04 STEP 1271 H=1271:00 +CPU= 21.098
  18:54:42 STEP 1272 H=1272:00 +CPU= 36.057
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001272
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001272
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001272
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001248
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001248
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001248
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001260
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001260
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001260
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_52.zip
Uploading the intermediate file: upload_file_52.zip
  18:55:03 STEP 1273 H=1273:00 +CPU= 20.021
  18:55:24 STEP 1274 H=1274:00 +CPU= 20.451
  18:55:57 STEP 1275 H=1275:00 +CPU= 30.864
  18:56:20 STEP 1276 H=1276:00 +CPU= 21.460
  18:56:43 STEP 1277 H=1277:00 +CPU= 20.921
  18:57:16 STEP 1278 H=1278:00 +CPU= 31.438
  18:57:36 STEP 1279 H=1279:00 +CPU= 19.772
  18:57:57 STEP 1280 H=1280:00 +CPU= 19.957
  18:58:27 STEP 1281 H=1281:00 +CPU= 29.345
  18:58:47 STEP 1282 H=1282:00 +CPU= 19.515
  18:59:07 STEP 1283 H=1283:00 +CPU= 19.263
  18:59:40 STEP 1284 H=1284:00 +CPU= 32.420
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001284
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001284
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001284
  19:00:00 STEP 1285 H=1285:00 +CPU= 19.420
  19:00:20 STEP 1286 H=1286:00 +CPU= 19.759
  19:00:51 STEP 1287 H=1287:00 +CPU= 29.758
  19:01:11 STEP 1288 H=1288:00 +CPU= 19.959
  19:01:31 STEP 1289 H=1289:00 +CPU= 19.779
  19:02:04 STEP 1290 H=1290:00 +CPU= 31.157
  19:02:24 STEP 1291 H=1291:00 +CPU= 19.768
  19:02:45 STEP 1292 H=1292:00 +CPU= 20.600
  19:03:17 STEP 1293 H=1293:00 +CPU= 30.751
  19:03:38 STEP 1294 H=1294:00 +CPU= 20.717
  19:04:00 STEP 1295 H=1295:00 +CPU= 21.439
  19:04:39 STEP 1296 H=1296:00 +CPU= 36.959
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001296
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001296
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001296
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001272
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001272
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001272
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001284
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001284
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001284
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_53.zip
Uploading the intermediate file: upload_file_53.zip
  19:05:01 STEP 1297 H=1297:00 +CPU= 20.493
  19:05:22 STEP 1298 H=1298:00 +CPU= 20.613
  19:05:54 STEP 1299 H=1299:00 +CPU= 30.887
  19:06:17 STEP 1300 H=1300:00 +CPU= 21.411
  19:06:36 STEP 1301 H=1301:00 +CPU= 19.161
  19:07:07 STEP 1302 H=1302:00 +CPU= 29.990
  19:07:28 STEP 1303 H=1303:00 +CPU= 19.865
  19:07:49 STEP 1304 H=1304:00 +CPU= 19.867
  19:08:19 STEP 1305 H=1305:00 +CPU= 29.811
  19:08:41 STEP 1306 H=1306:00 +CPU= 20.397
  19:09:02 STEP 1307 H=1307:00 +CPU= 19.863
  19:09:36 STEP 1308 H=1308:00 +CPU= 32.959
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001308
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001308
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001308
  19:09:57 STEP 1309 H=1309:00 +CPU= 20.252
  19:10:19 STEP 1310 H=1310:00 +CPU= 21.013
  19:10:50 STEP 1311 H=1311:00 +CPU= 30.578
  19:11:11 STEP 1312 H=1312:00 +CPU= 20.483
  19:11:31 STEP 1313 H=1313:00 +CPU= 20.030
  19:12:03 STEP 1314 H=1314:00 +CPU= 30.749
  19:12:24 STEP 1315 H=1315:00 +CPU= 20.097
  19:12:44 STEP 1316 H=1316:00 +CPU= 20.182
  19:13:15 STEP 1317 H=1317:00 +CPU= 30.517
  19:13:35 STEP 1318 H=1318:00 +CPU= 20.208
  19:13:56 STEP 1319 H=1319:00 +CPU= 20.733
  19:14:32 STEP 1320 H=1320:00 +CPU= 33.151
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001320
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001320
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001320
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001296
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001296
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001296
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001308
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001308
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001308
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_54.zip
Uploading the intermediate file: upload_file_54.zip
  19:14:52 STEP 1321 H=1321:00 +CPU= 19.627
  19:15:12 STEP 1322 H=1322:00 +CPU= 19.354
  19:15:43 STEP 1323 H=1323:00 +CPU= 31.016
  19:16:04 STEP 1324 H=1324:00 +CPU= 20.675
  19:16:22 STEP 1325 H=1325:00 +CPU= 17.294
  19:16:50 STEP 1326 H=1326:00 +CPU= 27.824
  19:17:08 STEP 1327 H=1327:00 +CPU= 17.689
  19:17:26 STEP 1328 H=1328:00 +CPU= 17.719
  19:17:59 STEP 1329 H=1329:00 +CPU= 30.826
  19:18:19 STEP 1330 H=1330:00 +CPU= 19.948
  19:18:35 STEP 1331 H=1331:00 +CPU= 15.991
  19:19:07 STEP 1332 H=1332:00 +CPU= 31.808
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001332
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001332
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001332
  19:19:27 STEP 1333 H=1333:00 +CPU= 19.437
  19:19:45 STEP 1334 H=1334:00 +CPU= 17.949
  19:20:14 STEP 1335 H=1335:00 +CPU= 28.781
  19:20:34 STEP 1336 H=1336:00 +CPU= 19.769
  19:20:52 STEP 1337 H=1337:00 +CPU= 18.346
  19:21:22 STEP 1338 H=1338:00 +CPU= 29.815
  19:21:42 STEP 1339 H=1339:00 +CPU= 19.984
  19:22:02 STEP 1340 H=1340:00 +CPU= 19.863
  19:22:32 STEP 1341 H=1341:00 +CPU= 28.780
  19:22:53 STEP 1342 H=1342:00 +CPU= 20.930
  19:23:12 STEP 1343 H=1343:00 +CPU= 19.351
  19:23:48 STEP 1344 H=1344:00 +CPU= 34.495
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001344
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001344
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001344
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001320
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001320
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001320
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001332
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001332
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001332
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_55.zip
Uploading the intermediate file: upload_file_55.zip
  19:24:07 STEP 1345 H=1345:00 +CPU= 19.444
  19:24:27 STEP 1346 H=1346:00 +CPU= 19.250
  19:24:56 STEP 1347 H=1347:00 +CPU= 28.939
  19:25:17 STEP 1348 H=1348:00 +CPU= 20.263
  19:25:36 STEP 1349 H=1349:00 +CPU= 19.546
  19:26:07 STEP 1350 H=1350:00 +CPU= 30.649
  19:26:27 STEP 1351 H=1351:00 +CPU= 19.425
  19:26:47 STEP 1352 H=1352:00 +CPU= 19.972
  19:27:18 STEP 1353 H=1353:00 +CPU= 30.608
  19:27:38 STEP 1354 H=1354:00 +CPU= 20.020
  19:27:58 STEP 1355 H=1355:00 +CPU= 19.515
  19:28:32 STEP 1356 H=1356:00 +CPU= 33.743
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001356
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001356
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001356
  19:28:52 STEP 1357 H=1357:00 +CPU= 19.487
  19:29:12 STEP 1358 H=1358:00 +CPU= 20.754
  19:29:45 STEP 1359 H=1359:00 +CPU= 32.143
  19:30:04 STEP 1360 H=1360:00 +CPU= 19.590
  19:30:23 STEP 1361 H=1361:00 +CPU= 18.824
  19:30:53 STEP 1362 H=1362:00 +CPU= 30.066
  19:31:12 STEP 1363 H=1363:00 +CPU= 18.535
  19:31:31 STEP 1364 H=1364:00 +CPU= 18.956
  19:32:00 STEP 1365 H=1365:00 +CPU= 28.983
  19:32:19 STEP 1366 H=1366:00 +CPU= 18.544
  19:32:38 STEP 1367 H=1367:00 +CPU= 18.959
  19:33:12 STEP 1368 H=1368:00 +CPU= 33.805
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001368
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001368
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001368
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001344
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001344
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001344
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001356
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001356
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001356
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_56.zip
Uploading the intermediate file: upload_file_56.zip
  19:33:30 STEP 1369 H=1369:00 +CPU= 18.606
  19:33:50 STEP 1370 H=1370:00 +CPU= 19.241
  19:34:20 STEP 1371 H=1371:00 +CPU= 29.100
  19:34:40 STEP 1372 H=1372:00 +CPU= 19.567
  19:34:58 STEP 1373 H=1373:00 +CPU= 18.117
  19:35:29 STEP 1374 H=1374:00 +CPU= 30.483
  19:35:49 STEP 1375 H=1375:00 +CPU= 19.543
  19:36:10 STEP 1376 H=1376:00 +CPU= 19.543
  19:36:41 STEP 1377 H=1377:00 +CPU= 30.135
  19:37:01 STEP 1378 H=1378:00 +CPU= 19.951
  19:37:21 STEP 1379 H=1379:00 +CPU= 19.089
  19:37:54 STEP 1380 H=1380:00 +CPU= 32.521
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001380
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001380
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001380
  19:38:14 STEP 1381 H=1381:00 +CPU= 19.374
  19:38:33 STEP 1382 H=1382:00 +CPU= 19.049
  19:39:03 STEP 1383 H=1383:00 +CPU= 29.202
  19:39:23 STEP 1384 H=1384:00 +CPU= 20.116
  19:39:42 STEP 1385 H=1385:00 +CPU= 18.753
  19:40:12 STEP 1386 H=1386:00 +CPU= 29.041
  19:40:31 STEP 1387 H=1387:00 +CPU= 19.279
  19:40:51 STEP 1388 H=1388:00 +CPU= 19.439
  19:41:20 STEP 1389 H=1389:00 +CPU= 28.792
  19:41:40 STEP 1390 H=1390:00 +CPU= 19.635
  19:42:00 STEP 1391 H=1391:00 +CPU= 19.612
  19:42:36 STEP 1392 H=1392:00 +CPU= 35.864
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001392
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001392
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001392
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001368
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001368
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001368
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001380
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001380
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001380
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_57.zip
Uploading the intermediate file: upload_file_57.zip
  19:42:56 STEP 1393 H=1393:00 +CPU= 19.730
  19:43:15 STEP 1394 H=1394:00 +CPU= 19.371
  19:43:45 STEP 1395 H=1395:00 +CPU= 29.419
  19:44:04 STEP 1396 H=1396:00 +CPU= 19.351
  19:44:25 STEP 1397 H=1397:00 +CPU= 20.863
  19:44:56 STEP 1398 H=1398:00 +CPU= 31.240
  19:45:16 STEP 1399 H=1399:00 +CPU= 19.415
  19:45:36 STEP 1400 H=1400:00 +CPU= 19.917
  19:46:06 STEP 1401 H=1401:00 +CPU= 30.337
  19:46:27 STEP 1402 H=1402:00 +CPU= 20.754
  19:46:48 STEP 1403 H=1403:00 +CPU= 20.343
  19:47:22 STEP 1404 H=1404:00 +CPU= 33.851
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001404
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001404
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001404
  19:47:41 STEP 1405 H=1405:00 +CPU= 19.650
  19:48:00 STEP 1406 H=1406:00 +CPU= 18.861
  19:48:28 STEP 1407 H=1407:00 +CPU= 28.048
  19:48:47 STEP 1408 H=1408:00 +CPU= 18.310
  19:49:05 STEP 1409 H=1409:00 +CPU= 18.356
  19:49:34 STEP 1410 H=1410:00 +CPU= 29.001
  19:49:54 STEP 1411 H=1411:00 +CPU= 19.038
  19:50:13 STEP 1412 H=1412:00 +CPU= 19.306
  19:50:44 STEP 1413 H=1413:00 +CPU= 30.679
  19:51:05 STEP 1414 H=1414:00 +CPU= 20.541
  19:51:25 STEP 1415 H=1415:00 +CPU= 20.423
  19:52:00 STEP 1416 H=1416:00 +CPU= 34.732
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001416
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001416
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001416
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001392
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001392
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001392
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001404
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001404
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001404
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_58.zip
Uploading the intermediate file: upload_file_58.zip
  19:52:20 STEP 1417 H=1417:00 +CPU= 19.554
Uploading trickle at timestep: 5097600
  19:52:40 STEP 1418 H=1418:00 +CPU= 19.693
  19:53:11 STEP 1419 H=1419:00 +CPU= 30.160
  19:53:32 STEP 1420 H=1420:00 +CPU= 20.896
  19:53:51 STEP 1421 H=1421:00 +CPU= 19.587
  19:54:21 STEP 1422 H=1422:00 +CPU= 29.802
  19:54:41 STEP 1423 H=1423:00 +CPU= 19.766
  19:55:01 STEP 1424 H=1424:00 +CPU= 19.711
  19:55:31 STEP 1425 H=1425:00 +CPU= 30.164
  19:55:51 STEP 1426 H=1426:00 +CPU= 19.824
  19:56:11 STEP 1427 H=1427:00 +CPU= 19.131
  19:56:43 STEP 1428 H=1428:00 +CPU= 32.110
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001428
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001428
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001428
  19:57:03 STEP 1429 H=1429:00 +CPU= 19.306
  19:57:22 STEP 1430 H=1430:00 +CPU= 19.067
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0006
(argv4) batchid: 987
(argv5) wuid: 12203650
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/home/boinc/slots/10
Project directory is: /mnt/home/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650
..mkdir for temp folder for results failed
Copying: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/home/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/home/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d4cc118baf285b3edee01c230448a2d3 to: /mnt/home/boinc/slots/10/oifs_43r3_ps_0006_2018050100_123_987_12203650.zip
Unzipping the namelist zip file: /mnt/home/boinc/slots/10/oifs_43r3_ps_0006_2018050100_123_987_12203650.zip
ic_ancil_file: ic_ancil_12203650
ifsdata_file: ifsdata_12203650
climate_data_file: clim_data_12203650
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2eb215ff8b4befb44926d7fc2f1a9295 to: /mnt/home/boinc/slots/10/ic_ancil_12203650.zip
Unzipping the IC ancils zip file: /mnt/home/boinc/slots/10/ic_ancil_12203650.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e5de5e7c9c0ffd484b6cd17c66e9be41 to: /mnt/home/boinc/slots/10/ifsdata/ifsdata_12203650.zip
Unzipping the ifsdata_zip file: /mnt/home/boinc/slots/10/ifsdata/ifsdata_12203650.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/home/boinc/slots/10/159l_2/clim_data_12203650.zip
Unzipping the climate data zip file: /mnt/home/boinc/slots/10/159l_2/clim_data_12203650.zip
Checking for progress XML file: /mnt/home/boinc/slots/10/progress_file_12203650.xml
Opened progress file ok : /mnt/home/boinc/slots/10/progress_file_12203650.xml
-- Model is restarting --
Adjusting last_iter, 1431, to previous model restart step.
Creating progress file: /mnt/home/boinc/slots/10/progress_file_12203650.xml
last_cpu_time: 36486
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973
The child process has been launched with process id: 28930
Executing the command: /mnt/home/boinc/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:x570gamingx:1:1:28930:28930] [20230120:200936:1674241776:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:10:15 STEP 1416 H=1416:00 +CPU= 34.732
  20:10:36 STEP 1417 H=1417:00 +CPU= 20.468
  20:10:57 STEP 1418 H=1418:00 +CPU= 20.549
  20:11:30 STEP 1419 H=1419:00 +CPU= 31.552
  20:11:51 STEP 1420 H=1420:00 +CPU= 20.088
  20:12:10 STEP 1421 H=1421:00 +CPU= 19.189
  20:12:42 STEP 1422 H=1422:00 +CPU= 31.054
  20:13:02 STEP 1423 H=1423:00 +CPU= 19.580
  20:13:22 STEP 1424 H=1424:00 +CPU= 19.349
  20:13:53 STEP 1425 H=1425:00 +CPU= 29.972
  20:14:14 STEP 1426 H=1426:00 +CPU= 20.293
  20:14:34 STEP 1427 H=1427:00 +CPU= 19.898
  20:15:10 STEP 1428 H=1428:00 +CPU= 35.338
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001428
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001428
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001428
  20:15:31 STEP 1429 H=1429:00 +CPU= 20.203
  20:15:52 STEP 1430 H=1430:00 +CPU= 20.666
  20:16:24 STEP 1431 H=1431:00 +CPU= 31.035
  20:16:46 STEP 1432 H=1432:00 +CPU= 21.120
  20:17:06 STEP 1433 H=1433:00 +CPU= 20.156
  20:17:39 STEP 1434 H=1434:00 +CPU= 31.714
  20:18:00 STEP 1435 H=1435:00 +CPU= 20.285
  20:18:21 STEP 1436 H=1436:00 +CPU= 20.921
  20:18:53 STEP 1437 H=1437:00 +CPU= 30.891
  20:19:14 STEP 1438 H=1438:00 +CPU= 20.745
  20:19:34 STEP 1439 H=1439:00 +CPU= 19.212
  20:20:08 STEP 1440 H=1440:00 +CPU= 32.860
Moving to projects directory: /mnt/home/boinc/slots/10/ICMGGhq0f+001440
Moving to projects directory: /mnt/home/boinc/slots/10/ICMSHhq0f+001440
Moving to projects directory: /mnt/home/boinc/slots/10/ICMUAhq0f+001440
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001416
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001416
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001416
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMGGhq0f+001428
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMSHhq0f+001428
Adding to the zip: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_12203650/ICMUAhq0f+001428
Zipping up the intermediate file: /mnt/home/boinc/projects/climateprediction.net/oifs_43r3_ps_0006_2018050100_123_987_12203650_0_r1444631973_59.zip
Uploading the intermediate file: upload_file_59.zip
  20:20:26 STEP 1441 H=1441:00 +CPU= 17.599
  20:20:44 STEP 1442 H=1442:00 +CPU= 18.088
  20:21:12 STEP 1443 H=1443:00 +CPU= 27.310
  20:21:33 STEP 1444 H=1444:00 +CPU= 19.833
  20:21:53 STEP 1445 H=1445:00 +CPU= 19.485
  20:22:25 STEP 1446 H=1446:00 +CPU= 31.013
  20:22:44 STEP 1447 H=1447:00 +CPU= 17.920
  20:23:02 STEP 1448 H=1448:00 +CPU= 17.650
  20:23:28 STEP 1449 H=1449:00 +CPU= 25.890
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:31:30 1532028 22290117 oifs_43r3_ps_0006_2018050100_123_987_12203650_0 5,097,600 36,147 0.0071
09 May 2023 16:24:45 1532028 22290117 oifs_43r3_ps_0006_2018050100_123_987_12203650_0 1,468,800 10,410 0.0071


©2024 climateprediction.net