climateprediction.net home page
Task 22248579

Task 22248579

Name oifs_43r3_ps_2377_2021050100_123_947_12165466_1
Workunit 12165466
Created 30 Nov 2022, 12:04:09 UTC
Sent 30 Nov 2022, 12:33:42 UTC
Report deadline 30 Dec 2022, 12:33:42 UTC
Received 1 Dec 2022, 18:49:45 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1536643
Run time 12 hours 37 min 42 sec
CPU time 12 hours 35 min 36 sec
Validate state Invalid
Credit 14,152.32
Device peak FLOPS 6.95 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,632.29 MB
Peak swap size 4,974.27 MB
Peak disk usage 1,225.87 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
136:00 +CPU= 23.334
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002136
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002136
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002136
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002112
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002112
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002112
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002124
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002124
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002124
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_88.zip
Uploading the intermediate file: upload_file_88.zip
  16:14:15 STEP 2137 H=2137:00 +CPU= 12.506
  16:14:28 STEP 2138 H=2138:00 +CPU= 12.999
Uploading trickle at timestep: 7689600
  16:14:48 STEP 2139 H=2139:00 +CPU= 20.297
  16:15:01 STEP 2140 H=2140:00 +CPU= 13.020
  16:15:14 STEP 2141 H=2141:00 +CPU= 12.657
  16:15:35 STEP 2142 H=2142:00 +CPU= 20.801
  16:15:48 STEP 2143 H=2143:00 +CPU= 12.572
  16:16:01 STEP 2144 H=2144:00 +CPU= 13.089
  16:16:21 STEP 2145 H=2145:00 +CPU= 20.206
  16:16:34 STEP 2146 H=2146:00 +CPU= 13.532
  16:16:48 STEP 2147 H=2147:00 +CPU= 13.175
  16:17:10 STEP 2148 H=2148:00 +CPU= 22.897
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002148
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002148
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002148
  16:17:23 STEP 2149 H=2149:00 +CPU= 12.750
  16:17:38 STEP 2150 H=2150:00 +CPU= 14.343
  16:17:59 STEP 2151 H=2151:00 +CPU= 21.355
  16:18:12 STEP 2152 H=2152:00 +CPU= 13.099
  16:18:26 STEP 2153 H=2153:00 +CPU= 13.648
  16:18:46 STEP 2154 H=2154:00 +CPU= 20.808
  16:18:59 STEP 2155 H=2155:00 +CPU= 12.538
  16:19:12 STEP 2156 H=2156:00 +CPU= 13.054
  16:19:32 STEP 2157 H=2157:00 +CPU= 20.198
  16:19:45 STEP 2158 H=2158:00 +CPU= 12.947
  16:19:58 STEP 2159 H=2159:00 +CPU= 12.992
  16:20:23 STEP 2160 H=2160:00 +CPU= 23.245
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002160
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002160
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002160
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002136
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002136
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002136
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002148
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002148
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002148
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_89.zip
Uploading the intermediate file: upload_file_89.zip
  16:20:36 STEP 2161 H=2161:00 +CPU= 12.562
Uploading trickle at timestep: 7776000
  16:20:49 STEP 2162 H=2162:00 +CPU= 13.027
  16:21:09 STEP 2163 H=2163:00 +CPU= 20.308
  16:21:22 STEP 2164 H=2164:00 +CPU= 12.982
  16:21:35 STEP 2165 H=2165:00 +CPU= 12.679
  16:21:56 STEP 2166 H=2166:00 +CPU= 20.856
  16:22:08 STEP 2167 H=2167:00 +CPU= 12.597
  16:22:22 STEP 2168 H=2168:00 +CPU= 13.205
  16:22:42 STEP 2169 H=2169:00 +CPU= 20.114
  16:22:55 STEP 2170 H=2170:00 +CPU= 12.888
  16:23:07 STEP 2171 H=2171:00 +CPU= 12.752
  16:23:30 STEP 2172 H=2172:00 +CPU= 22.418
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002172
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002172
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002172
  16:23:43 STEP 2173 H=2173:00 +CPU= 12.705
  16:23:56 STEP 2174 H=2174:00 +CPU= 13.546
  16:24:16 STEP 2175 H=2175:00 +CPU= 20.234
  16:24:29 STEP 2176 H=2176:00 +CPU= 13.010
  16:24:42 STEP 2177 H=2177:00 +CPU= 12.683
  16:25:03 STEP 2178 H=2178:00 +CPU= 20.664
  16:25:15 STEP 2179 H=2179:00 +CPU= 12.608
  16:25:28 STEP 2180 H=2180:00 +CPU= 13.112
  16:25:49 STEP 2181 H=2181:00 +CPU= 20.563
  16:26:03 STEP 2182 H=2182:00 +CPU= 13.951
  16:26:16 STEP 2183 H=2183:00 +CPU= 13.069
  16:26:39 STEP 2184 H=2184:00 +CPU= 23.347
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002184
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002184
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002184
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002160
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002160
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002160
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002172
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002172
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002172
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_90.zip
Uploading the intermediate file: upload_file_90.zip
  16:26:53 STEP 2185 H=2185:00 +CPU= 13.510
  16:27:08 STEP 2186 H=2186:00 +CPU= 15.422
Uploading trickle at timestep: 7862400
  16:27:30 STEP 2187 H=2187:00 +CPU= 21.468
  16:27:43 STEP 2188 H=2188:00 +CPU= 13.599
  16:27:57 STEP 2189 H=2189:00 +CPU= 13.649
  16:28:20 STEP 2190 H=2190:00 +CPU= 22.828
  16:28:33 STEP 2191 H=2191:00 +CPU= 12.785
  16:28:46 STEP 2192 H=2192:00 +CPU= 13.869
  16:29:09 STEP 2193 H=2193:00 +CPU= 22.445
  16:29:22 STEP 2194 H=2194:00 +CPU= 13.187
  16:29:35 STEP 2195 H=2195:00 +CPU= 12.632
  16:29:57 STEP 2196 H=2196:00 +CPU= 21.825
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002196
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002196
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002196
  16:30:09 STEP 2197 H=2197:00 +CPU= 12.133
  16:30:21 STEP 2198 H=2198:00 +CPU= 12.520
  16:30:41 STEP 2199 H=2199:00 +CPU= 19.537
  16:30:53 STEP 2200 H=2200:00 +CPU= 12.512
  16:31:05 STEP 2201 H=2201:00 +CPU= 12.229
  16:31:25 STEP 2202 H=2202:00 +CPU= 19.981
  16:31:38 STEP 2203 H=2203:00 +CPU= 12.144
  16:31:50 STEP 2204 H=2204:00 +CPU= 12.538
  16:32:10 STEP 2205 H=2205:00 +CPU= 19.599
  16:32:23 STEP 2206 H=2206:00 +CPU= 13.367
  16:32:37 STEP 2207 H=2207:00 +CPU= 14.238
  16:33:01 STEP 2208 H=2208:00 +CPU= 23.038
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002208
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002208
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002208
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002184
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002184
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002184
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002196
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002196
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002196
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_91.zip
Uploading the intermediate file: upload_file_91.zip
  16:33:14 STEP 2209 H=2209:00 +CPU= 12.155
  16:33:26 STEP 2210 H=2210:00 +CPU= 12.603
Uploading trickle at timestep: 7948800
  16:33:46 STEP 2211 H=2211:00 +CPU= 19.749
  16:34:00 STEP 2212 H=2212:00 +CPU= 13.765
  16:34:12 STEP 2213 H=2213:00 +CPU= 12.324
  16:34:32 STEP 2214 H=2214:00 +CPU= 20.182
  16:34:46 STEP 2215 H=2215:00 +CPU= 14.161
  16:34:59 STEP 2216 H=2216:00 +CPU= 12.635
  16:35:20 STEP 2217 H=2217:00 +CPU= 21.257
  16:35:33 STEP 2218 H=2218:00 +CPU= 13.002
  16:35:46 STEP 2219 H=2219:00 +CPU= 13.218
  16:36:10 STEP 2220 H=2220:00 +CPU= 23.806
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002220
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002220
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002220
  16:36:22 STEP 2221 H=2221:00 +CPU= 12.236
  16:36:35 STEP 2222 H=2222:00 +CPU= 12.582
  16:36:55 STEP 2223 H=2223:00 +CPU= 19.575
  16:37:07 STEP 2224 H=2224:00 +CPU= 12.557
  16:37:19 STEP 2225 H=2225:00 +CPU= 12.238
  16:37:40 STEP 2226 H=2226:00 +CPU= 20.134
  16:37:52 STEP 2227 H=2227:00 +CPU= 12.148
  16:38:04 STEP 2228 H=2228:00 +CPU= 12.525
  16:38:24 STEP 2229 H=2229:00 +CPU= 19.534
  16:38:36 STEP 2230 H=2230:00 +CPU= 12.511
  16:38:49 STEP 2231 H=2231:00 +CPU= 12.489
  16:39:11 STEP 2232 H=2232:00 +CPU= 22.551
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002232
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002232
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002232
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002208
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002208
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002208
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002220
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002220
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002220
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_92.zip
Uploading the intermediate file: upload_file_92.zip
  16:39:23 STEP 2233 H=2233:00 +CPU= 12.132
  16:39:36 STEP 2234 H=2234:00 +CPU= 12.531
Uploading trickle at timestep: 8035200
  16:39:56 STEP 2235 H=2235:00 +CPU= 19.590
  16:40:08 STEP 2236 H=2236:00 +CPU= 12.577
  16:40:20 STEP 2237 H=2237:00 +CPU= 12.115
  16:40:40 STEP 2238 H=2238:00 +CPU= 19.926
  16:40:52 STEP 2239 H=2239:00 +CPU= 12.134
  16:41:05 STEP 2240 H=2240:00 +CPU= 12.547
  16:41:25 STEP 2241 H=2241:00 +CPU= 19.680
  16:41:37 STEP 2242 H=2242:00 +CPU= 12.570
  16:41:49 STEP 2243 H=2243:00 +CPU= 12.141
  16:42:11 STEP 2244 H=2244:00 +CPU= 21.621
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002244
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002244
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002244
  16:42:23 STEP 2245 H=2245:00 +CPU= 12.184
  16:42:36 STEP 2246 H=2246:00 +CPU= 12.559
  16:42:55 STEP 2247 H=2247:00 +CPU= 19.793
  16:43:08 STEP 2248 H=2248:00 +CPU= 12.671
  16:43:22 STEP 2249 H=2249:00 +CPU= 14.108
  16:43:42 STEP 2250 H=2250:00 +CPU= 20.113
  16:43:54 STEP 2251 H=2251:00 +CPU= 12.156
  16:44:07 STEP 2252 H=2252:00 +CPU= 12.535
  16:44:27 STEP 2253 H=2253:00 +CPU= 19.530
  16:44:39 STEP 2254 H=2254:00 +CPU= 12.565
  16:44:52 STEP 2255 H=2255:00 +CPU= 12.563
  16:45:14 STEP 2256 H=2256:00 +CPU= 22.489
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002256
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002256
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002256
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002232
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002232
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002232
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002244
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002244
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002244
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_93.zip
Uploading the intermediate file: upload_file_93.zip
  16:45:26 STEP 2257 H=2257:00 +CPU= 12.131
  16:45:39 STEP 2258 H=2258:00 +CPU= 12.520
Uploading trickle at timestep: 8121600
  16:45:59 STEP 2259 H=2259:00 +CPU= 19.707
  16:46:11 STEP 2260 H=2260:00 +CPU= 12.529
  16:46:23 STEP 2261 H=2261:00 +CPU= 12.259
  16:46:43 STEP 2262 H=2262:00 +CPU= 20.096
  16:46:56 STEP 2263 H=2263:00 +CPU= 12.171
  16:47:08 STEP 2264 H=2264:00 +CPU= 12.517
  16:47:28 STEP 2265 H=2265:00 +CPU= 19.571
  16:47:40 STEP 2266 H=2266:00 +CPU= 12.569
  16:47:52 STEP 2267 H=2267:00 +CPU= 12.154
  16:48:14 STEP 2268 H=2268:00 +CPU= 21.755
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002268
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002268
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002268
  16:48:26 STEP 2269 H=2269:00 +CPU= 12.210
  16:48:39 STEP 2270 H=2270:00 +CPU= 12.618
  16:48:59 STEP 2271 H=2271:00 +CPU= 20.012
  16:49:13 STEP 2272 H=2272:00 +CPU= 13.780
  16:49:25 STEP 2273 H=2273:00 +CPU= 12.193
  16:49:45 STEP 2274 H=2274:00 +CPU= 20.115
  16:49:57 STEP 2275 H=2275:00 +CPU= 12.144
  16:50:10 STEP 2276 H=2276:00 +CPU= 12.995
  16:50:31 STEP 2277 H=2277:00 +CPU= 20.666
  16:50:43 STEP 2278 H=2278:00 +CPU= 12.567
  16:50:57 STEP 2279 H=2279:00 +CPU= 13.616
  16:51:20 STEP 2280 H=2280:00 +CPU= 22.550
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002280
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002280
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002280
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002256
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002256
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002256
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002268
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002268
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002268
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_94.zip
Uploading the intermediate file: upload_file_94.zip
  16:51:35 STEP 2281 H=2281:00 +CPU= 15.111
Uploading trickle at timestep: 8208000
  16:51:48 STEP 2282 H=2282:00 +CPU= 13.511
  16:52:09 STEP 2283 H=2283:00 +CPU= 20.271
  16:52:23 STEP 2284 H=2284:00 +CPU= 14.355
  16:52:35 STEP 2285 H=2285:00 +CPU= 12.149
  16:52:55 STEP 2286 H=2286:00 +CPU= 19.988
  16:53:09 STEP 2287 H=2287:00 +CPU= 13.501
  16:53:22 STEP 2288 H=2288:00 +CPU= 13.850
  16:53:42 STEP 2289 H=2289:00 +CPU= 19.572
  16:53:54 STEP 2290 H=2290:00 +CPU= 12.542
  16:54:07 STEP 2291 H=2291:00 +CPU= 12.156
  16:54:28 STEP 2292 H=2292:00 +CPU= 21.771
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002292
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002292
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002292
  16:54:41 STEP 2293 H=2293:00 +CPU= 12.148
  16:54:53 STEP 2294 H=2294:00 +CPU= 12.797
  16:55:13 STEP 2295 H=2295:00 +CPU= 19.659
  16:55:26 STEP 2296 H=2296:00 +CPU= 12.560
  16:55:38 STEP 2297 H=2297:00 +CPU= 12.180
  16:55:58 STEP 2298 H=2298:00 +CPU= 19.974
  16:56:10 STEP 2299 H=2299:00 +CPU= 12.144
  16:56:22 STEP 2300 H=2300:00 +CPU= 12.560
  16:56:45 STEP 2301 H=2301:00 +CPU= 22.296
  16:57:04 STEP 2302 H=2302:00 +CPU= 18.817
  16:57:20 STEP 2303 H=2303:00 +CPU= 16.251
  16:57:48 STEP 2304 H=2304:00 +CPU= 28.296
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002304
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002304
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002304
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002280
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002280
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002280
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002292
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002292
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002292
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_95.zip
Uploading the intermediate file: upload_file_95.zip
  16:58:04 STEP 2305 H=2305:00 +CPU= 15.587
Uploading trickle at timestep: 8294400
  16:58:22 STEP 2306 H=2306:00 +CPU= 18.544
  16:58:52 STEP 2307 H=2307:00 +CPU= 29.973
  16:59:10 STEP 2308 H=2308:00 +CPU= 17.839
  16:59:26 STEP 2309 H=2309:00 +CPU= 15.405
  16:59:51 STEP 2310 H=2310:00 +CPU= 25.519
  17:00:07 STEP 2311 H=2311:00 +CPU= 15.661
  17:00:22 STEP 2312 H=2312:00 +CPU= 15.713
  17:00:47 STEP 2313 H=2313:00 +CPU= 24.637
  17:01:03 STEP 2314 H=2314:00 +CPU= 15.744
  17:01:18 STEP 2315 H=2315:00 +CPU= 15.244
  17:01:43 STEP 2316 H=2316:00 +CPU= 24.793
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002316
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002316
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002316
  17:01:55 STEP 2317 H=2317:00 +CPU= 12.158
  17:02:11 STEP 2318 H=2318:00 +CPU= 15.914
  17:02:37 STEP 2319 H=2319:00 +CPU= 25.869
  17:02:53 STEP 2320 H=2320:00 +CPU= 15.735
  17:03:09 STEP 2321 H=2321:00 +CPU= 16.746
  17:03:37 STEP 2322 H=2322:00 +CPU= 27.618
  17:03:53 STEP 2323 H=2323:00 +CPU= 15.719
  17:04:08 STEP 2324 H=2324:00 +CPU= 15.639
  17:04:33 STEP 2325 H=2325:00 +CPU= 24.888
  17:04:49 STEP 2326 H=2326:00 +CPU= 15.722
  17:05:05 STEP 2327 H=2327:00 +CPU= 16.131
  17:05:38 STEP 2328 H=2328:00 +CPU= 28.198
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002328
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002328
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002328
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002304
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002304
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002304
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002316
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002316
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002316
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_96.zip
Uploading the intermediate file: upload_file_96.zip
  17:05:53 STEP 2329 H=2329:00 +CPU= 15.196
Uploading trickle at timestep: 8380800
  17:06:09 STEP 2330 H=2330:00 +CPU= 15.436
  17:06:33 STEP 2331 H=2331:00 +CPU= 24.362
  17:06:52 STEP 2332 H=2332:00 +CPU= 18.658
  17:07:08 STEP 2333 H=2333:00 +CPU= 16.099
  17:07:33 STEP 2334 H=2334:00 +CPU= 25.134
  17:07:48 STEP 2335 H=2335:00 +CPU= 15.272
  17:08:04 STEP 2336 H=2336:00 +CPU= 15.954
  17:08:29 STEP 2337 H=2337:00 +CPU= 25.046
  17:08:46 STEP 2338 H=2338:00 +CPU= 16.595
  17:09:01 STEP 2339 H=2339:00 +CPU= 15.556
  17:09:28 STEP 2340 H=2340:00 +CPU= 27.069
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002340
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002340
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002340
  17:09:44 STEP 2341 H=2341:00 +CPU= 15.427
  17:10:00 STEP 2342 H=2342:00 +CPU= 16.538
  17:10:20 STEP 2343 H=2343:00 +CPU= 19.649
  17:10:33 STEP 2344 H=2344:00 +CPU= 12.856
  17:10:46 STEP 2345 H=2345:00 +CPU= 13.085
  17:11:08 STEP 2346 H=2346:00 +CPU= 21.833
  17:11:21 STEP 2347 H=2347:00 +CPU= 12.965
  17:11:33 STEP 2348 H=2348:00 +CPU= 12.550
  17:11:53 STEP 2349 H=2349:00 +CPU= 19.616
  17:12:06 STEP 2350 H=2350:00 +CPU= 12.549
  17:12:19 STEP 2351 H=2351:00 +CPU= 13.301
  17:12:42 STEP 2352 H=2352:00 +CPU= 23.626
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002352
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002352
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002352
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002328
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002328
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002328
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002340
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002340
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002340
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_97.zip
Uploading the intermediate file: upload_file_97.zip
  17:12:55 STEP 2353 H=2353:00 +CPU= 12.543
Uploading trickle at timestep: 8467200
  17:13:09 STEP 2354 H=2354:00 +CPU= 13.725
  17:13:29 STEP 2355 H=2355:00 +CPU= 20.175
  17:13:42 STEP 2356 H=2356:00 +CPU= 12.630
  17:13:54 STEP 2357 H=2357:00 +CPU= 12.176
  17:14:14 STEP 2358 H=2358:00 +CPU= 20.119
  17:14:26 STEP 2359 H=2359:00 +CPU= 12.199
  17:14:39 STEP 2360 H=2360:00 +CPU= 12.603
  17:14:58 STEP 2361 H=2361:00 +CPU= 19.693
  17:15:11 STEP 2362 H=2362:00 +CPU= 12.636
  17:15:23 STEP 2363 H=2363:00 +CPU= 12.229
  17:15:45 STEP 2364 H=2364:00 +CPU= 21.846
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002364
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002364
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002364
  17:15:57 STEP 2365 H=2365:00 +CPU= 12.213
  17:16:10 STEP 2366 H=2366:00 +CPU= 12.645
  17:16:30 STEP 2367 H=2367:00 +CPU= 19.647
  17:16:42 STEP 2368 H=2368:00 +CPU= 12.602
  17:16:54 STEP 2369 H=2369:00 +CPU= 12.156
  17:17:14 STEP 2370 H=2370:00 +CPU= 20.159
  17:17:27 STEP 2371 H=2371:00 +CPU= 12.238
  17:17:39 STEP 2372 H=2372:00 +CPU= 12.719
  17:18:00 STEP 2373 H=2373:00 +CPU= 20.106
  17:18:12 STEP 2374 H=2374:00 +CPU= 12.781
  17:18:25 STEP 2375 H=2375:00 +CPU= 12.624
  17:18:48 STEP 2376 H=2376:00 +CPU= 22.639
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002376
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002376
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002376
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002352
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002352
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002352
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002364
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002364
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002364
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_98.zip
Uploading the intermediate file: upload_file_98.zip
  17:19:00 STEP 2377 H=2377:00 +CPU= 12.164
  17:19:12 STEP 2378 H=2378:00 +CPU= 12.610
Uploading trickle at timestep: 8553600
  17:19:32 STEP 2379 H=2379:00 +CPU= 19.716
  17:19:45 STEP 2380 H=2380:00 +CPU= 12.578
  17:19:57 STEP 2381 H=2381:00 +CPU= 12.670
  17:20:18 STEP 2382 H=2382:00 +CPU= 20.212
  17:20:30 STEP 2383 H=2383:00 +CPU= 12.249
  17:20:42 STEP 2384 H=2384:00 +CPU= 12.584
  17:21:02 STEP 2385 H=2385:00 +CPU= 19.726
  17:21:15 STEP 2386 H=2386:00 +CPU= 12.654
  17:21:27 STEP 2387 H=2387:00 +CPU= 12.248
  17:21:49 STEP 2388 H=2388:00 +CPU= 21.830
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002388
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002388
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002388
  17:22:01 STEP 2389 H=2389:00 +CPU= 12.267
  17:22:14 STEP 2390 H=2390:00 +CPU= 12.598
  17:22:33 STEP 2391 H=2391:00 +CPU= 19.715
  17:22:46 STEP 2392 H=2392:00 +CPU= 12.719
  17:22:58 STEP 2393 H=2393:00 +CPU= 12.272
  17:23:19 STEP 2394 H=2394:00 +CPU= 20.212
  17:23:31 STEP 2395 H=2395:00 +CPU= 12.292
  17:23:44 STEP 2396 H=2396:00 +CPU= 12.712
  17:24:03 STEP 2397 H=2397:00 +CPU= 19.698
  17:24:16 STEP 2398 H=2398:00 +CPU= 12.533
  17:24:28 STEP 2399 H=2399:00 +CPU= 12.513
  17:24:51 STEP 2400 H=2400:00 +CPU= 22.607
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002400
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002400
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002400
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002376
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002376
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002376
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002388
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002388
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002388
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_99.zip
Uploading the intermediate file: upload_file_99.zip
  17:25:03 STEP 2401 H=2401:00 +CPU= 12.204
  17:25:16 STEP 2402 H=2402:00 +CPU= 12.587
Uploading trickle at timestep: 8640000
  17:25:35 STEP 2403 H=2403:00 +CPU= 19.674
  17:25:48 STEP 2404 H=2404:00 +CPU= 12.558
  17:26:00 STEP 2405 H=2405:00 +CPU= 12.220
  17:26:23 STEP 2406 H=2406:00 +CPU= 22.359
  17:26:35 STEP 2407 H=2407:00 +CPU= 12.165
  17:26:47 STEP 2408 H=2408:00 +CPU= 12.632
  17:27:07 STEP 2409 H=2409:00 +CPU= 19.646
  17:27:20 STEP 2410 H=2410:00 +CPU= 12.627
  17:27:33 STEP 2411 H=2411:00 +CPU= 12.824
  17:27:56 STEP 2412 H=2412:00 +CPU= 23.373
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002412
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002412
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002412
  17:28:09 STEP 2413 H=2413:00 +CPU= 13.225
  17:28:23 STEP 2414 H=2414:00 +CPU= 13.609
  17:28:44 STEP 2415 H=2415:00 +CPU= 21.079
  17:28:57 STEP 2416 H=2416:00 +CPU= 13.593
  17:29:11 STEP 2417 H=2417:00 +CPU= 13.180
  17:29:32 STEP 2418 H=2418:00 +CPU= 21.545
  17:29:45 STEP 2419 H=2419:00 +CPU= 13.140
  17:29:59 STEP 2420 H=2420:00 +CPU= 13.602
  17:30:20 STEP 2421 H=2421:00 +CPU= 21.147
  17:30:34 STEP 2422 H=2422:00 +CPU= 13.591
  17:30:47 STEP 2423 H=2423:00 +CPU= 13.583
  17:31:12 STEP 2424 H=2424:00 +CPU= 24.348
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002424
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002424
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002424
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002400
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002400
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002400
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002412
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002412
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002412
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_100.zip
Uploading the intermediate file: upload_file_100.zip
  17:31:25 STEP 2425 H=2425:00 +CPU= 13.450
  17:31:39 STEP 2426 H=2426:00 +CPU= 13.568
Uploading trickle at timestep: 8726400
  17:32:00 STEP 2427 H=2427:00 +CPU= 21.090
  17:32:13 STEP 2428 H=2428:00 +CPU= 13.602
  17:32:26 STEP 2429 H=2429:00 +CPU= 13.192
  17:32:48 STEP 2430 H=2430:00 +CPU= 21.561
  17:33:01 STEP 2431 H=2431:00 +CPU= 13.177
  17:33:15 STEP 2432 H=2432:00 +CPU= 13.618
  17:33:36 STEP 2433 H=2433:00 +CPU= 21.066
  17:33:49 STEP 2434 H=2434:00 +CPU= 13.605
  17:34:03 STEP 2435 H=2435:00 +CPU= 13.180
  17:34:26 STEP 2436 H=2436:00 +CPU= 23.297
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002436
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002436
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002436
  17:34:39 STEP 2437 H=2437:00 +CPU= 13.212
  17:34:53 STEP 2438 H=2438:00 +CPU= 13.634
  17:35:14 STEP 2439 H=2439:00 +CPU= 21.066
  17:35:27 STEP 2440 H=2440:00 +CPU= 13.613
  17:35:41 STEP 2441 H=2441:00 +CPU= 13.187
  17:36:02 STEP 2442 H=2442:00 +CPU= 21.508
  17:36:15 STEP 2443 H=2443:00 +CPU= 13.180
  17:36:29 STEP 2444 H=2444:00 +CPU= 13.624
  17:36:50 STEP 2445 H=2445:00 +CPU= 21.060
  17:37:04 STEP 2446 H=2446:00 +CPU= 13.579
  17:37:17 STEP 2447 H=2447:00 +CPU= 13.570
  17:37:42 STEP 2448 H=2448:00 +CPU= 24.361
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002448
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002448
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002448
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002424
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002424
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002424
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002436
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002436
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002436
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_101.zip
Uploading the intermediate file: upload_file_101.zip
  17:37:55 STEP 2449 H=2449:00 +CPU= 13.147
  17:38:08 STEP 2450 H=2450:00 +CPU= 13.599
Uploading trickle at timestep: 8812800
  17:38:29 STEP 2451 H=2451:00 +CPU= 21.057
  17:38:43 STEP 2452 H=2452:00 +CPU= 13.603
  17:38:56 STEP 2453 H=2453:00 +CPU= 13.167
  17:39:18 STEP 2454 H=2454:00 +CPU= 21.560
  17:39:31 STEP 2455 H=2455:00 +CPU= 13.251
  17:39:45 STEP 2456 H=2456:00 +CPU= 13.631
  17:40:06 STEP 2457 H=2457:00 +CPU= 21.086
  17:40:19 STEP 2458 H=2458:00 +CPU= 13.611
  17:40:33 STEP 2459 H=2459:00 +CPU= 13.193
  17:40:56 STEP 2460 H=2460:00 +CPU= 23.262
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002460
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002460
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002460
  17:41:09 STEP 2461 H=2461:00 +CPU= 13.232
  17:41:23 STEP 2462 H=2462:00 +CPU= 13.618
  17:41:44 STEP 2463 H=2463:00 +CPU= 21.094
  17:41:57 STEP 2464 H=2464:00 +CPU= 13.595
  17:42:11 STEP 2465 H=2465:00 +CPU= 13.202
  17:42:32 STEP 2466 H=2466:00 +CPU= 21.548
  17:42:45 STEP 2467 H=2467:00 +CPU= 13.203
  17:42:59 STEP 2468 H=2468:00 +CPU= 13.628
  17:43:20 STEP 2469 H=2469:00 +CPU= 21.042
  17:43:34 STEP 2470 H=2470:00 +CPU= 13.610
  17:43:47 STEP 2471 H=2471:00 +CPU= 13.566
  17:44:11 STEP 2472 H=2472:00 +CPU= 24.313
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002472
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002472
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002472
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002448
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002448
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002448
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002460
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002460
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002460
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_102.zip
Uploading the intermediate file: upload_file_102.zip
  17:44:25 STEP 2473 H=2473:00 +CPU= 13.484
  17:44:38 STEP 2474 H=2474:00 +CPU= 13.584
Uploading trickle at timestep: 8899200
  17:45:00 STEP 2475 H=2475:00 +CPU= 21.163
  17:45:13 STEP 2476 H=2476:00 +CPU= 13.601
  17:45:26 STEP 2477 H=2477:00 +CPU= 13.241
  17:45:48 STEP 2478 H=2478:00 +CPU= 21.547
  17:46:01 STEP 2479 H=2479:00 +CPU= 13.234
  17:46:15 STEP 2480 H=2480:00 +CPU= 13.635
  17:46:36 STEP 2481 H=2481:00 +CPU= 21.137
  17:46:50 STEP 2482 H=2482:00 +CPU= 13.625
  17:47:03 STEP 2483 H=2483:00 +CPU= 13.244
  17:47:26 STEP 2484 H=2484:00 +CPU= 23.407
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002484
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002484
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002484
  17:47:40 STEP 2485 H=2485:00 +CPU= 13.201
  17:47:53 STEP 2486 H=2486:00 +CPU= 13.629
  17:48:14 STEP 2487 H=2487:00 +CPU= 21.062
  17:48:28 STEP 2488 H=2488:00 +CPU= 13.637
  17:48:41 STEP 2489 H=2489:00 +CPU= 13.225
  17:49:03 STEP 2490 H=2490:00 +CPU= 21.507
  17:49:16 STEP 2491 H=2491:00 +CPU= 13.151
  17:49:29 STEP 2492 H=2492:00 +CPU= 13.618
  17:49:50 STEP 2493 H=2493:00 +CPU= 21.053
  17:50:04 STEP 2494 H=2494:00 +CPU= 13.625
  17:50:18 STEP 2495 H=2495:00 +CPU= 13.582
  17:50:42 STEP 2496 H=2496:00 +CPU= 24.300
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002496
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002496
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002496
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002472
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002472
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002472
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002484
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002484
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002484
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_103.zip
Uploading the intermediate file: upload_file_103.zip
  17:50:55 STEP 2497 H=2497:00 +CPU= 13.195
Uploading trickle at timestep: 8985600
  17:51:09 STEP 2498 H=2498:00 +CPU= 13.591
  17:51:30 STEP 2499 H=2499:00 +CPU= 21.057
  17:51:43 STEP 2500 H=2500:00 +CPU= 13.622
  17:51:57 STEP 2501 H=2501:00 +CPU= 13.229
  17:52:18 STEP 2502 H=2502:00 +CPU= 21.511
  17:52:31 STEP 2503 H=2503:00 +CPU= 13.199
  17:52:45 STEP 2504 H=2504:00 +CPU= 13.670
  17:53:06 STEP 2505 H=2505:00 +CPU= 21.192
  17:53:20 STEP 2506 H=2506:00 +CPU= 13.636
  17:53:33 STEP 2507 H=2507:00 +CPU= 13.269
  17:53:56 STEP 2508 H=2508:00 +CPU= 23.328
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002508
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002508
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002508
  17:54:10 STEP 2509 H=2509:00 +CPU= 13.239
  17:54:23 STEP 2510 H=2510:00 +CPU= 13.673
  17:54:44 STEP 2511 H=2511:00 +CPU= 21.111
  17:54:58 STEP 2512 H=2512:00 +CPU= 13.609
  17:55:11 STEP 2513 H=2513:00 +CPU= 13.154
  17:55:33 STEP 2514 H=2514:00 +CPU= 21.625
  17:55:46 STEP 2515 H=2515:00 +CPU= 13.203
  17:56:00 STEP 2516 H=2516:00 +CPU= 13.578
  17:56:21 STEP 2517 H=2517:00 +CPU= 21.049
  17:56:34 STEP 2518 H=2518:00 +CPU= 13.628
  17:56:48 STEP 2519 H=2519:00 +CPU= 13.632
  17:57:12 STEP 2520 H=2520:00 +CPU= 24.284
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002520
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002520
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002520
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002496
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002496
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002496
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002508
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002508
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002508
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_104.zip
Uploading the intermediate file: upload_file_104.zip
  17:57:25 STEP 2521 H=2521:00 +CPU= 13.133
Uploading trickle at timestep: 9072000
  17:57:39 STEP 2522 H=2522:00 +CPU= 13.586
  17:58:00 STEP 2523 H=2523:00 +CPU= 20.992
  17:58:14 STEP 2524 H=2524:00 +CPU= 13.616
  17:58:27 STEP 2525 H=2525:00 +CPU= 13.177
  17:58:48 STEP 2526 H=2526:00 +CPU= 21.552
  17:59:01 STEP 2527 H=2527:00 +CPU= 13.196
  17:59:15 STEP 2528 H=2528:00 +CPU= 13.626
  17:59:36 STEP 2529 H=2529:00 +CPU= 21.007
  17:59:50 STEP 2530 H=2530:00 +CPU= 13.643
  18:00:03 STEP 2531 H=2531:00 +CPU= 13.216
  18:00:26 STEP 2532 H=2532:00 +CPU= 23.359
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002532
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002532
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002532
  18:00:40 STEP 2533 H=2533:00 +CPU= 13.205
  18:00:53 STEP 2534 H=2534:00 +CPU= 13.632
  18:01:14 STEP 2535 H=2535:00 +CPU= 21.268
  18:01:28 STEP 2536 H=2536:00 +CPU= 13.585
  18:01:41 STEP 2537 H=2537:00 +CPU= 13.222
  18:02:03 STEP 2538 H=2538:00 +CPU= 21.558
  18:02:16 STEP 2539 H=2539:00 +CPU= 13.312
  18:02:30 STEP 2540 H=2540:00 +CPU= 13.576
  18:02:51 STEP 2541 H=2541:00 +CPU= 21.061
  18:03:04 STEP 2542 H=2542:00 +CPU= 13.570
  18:03:18 STEP 2543 H=2543:00 +CPU= 13.556
  18:03:42 STEP 2544 H=2544:00 +CPU= 24.274
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002544
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002544
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002544
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002520
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002520
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002520
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002532
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002532
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002532
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_105.zip
Uploading the intermediate file: upload_file_105.zip
  18:03:55 STEP 2545 H=2545:00 +CPU= 13.201
Uploading trickle at timestep: 9158400
  18:04:09 STEP 2546 H=2546:00 +CPU= 13.551
  18:04:30 STEP 2547 H=2547:00 +CPU= 20.999
  18:04:43 STEP 2548 H=2548:00 +CPU= 13.560
  18:04:57 STEP 2549 H=2549:00 +CPU= 13.148
  18:05:18 STEP 2550 H=2550:00 +CPU= 21.541
  18:05:31 STEP 2551 H=2551:00 +CPU= 13.218
  18:05:45 STEP 2552 H=2552:00 +CPU= 13.602
  18:06:06 STEP 2553 H=2553:00 +CPU= 21.047
  18:06:20 STEP 2554 H=2554:00 +CPU= 13.633
  18:06:33 STEP 2555 H=2555:00 +CPU= 13.236
  18:06:56 STEP 2556 H=2556:00 +CPU= 23.222
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002556
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002556
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002556
  18:07:09 STEP 2557 H=2557:00 +CPU= 13.178
  18:07:23 STEP 2558 H=2558:00 +CPU= 13.578
  18:07:44 STEP 2559 H=2559:00 +CPU= 21.032
  18:07:57 STEP 2560 H=2560:00 +CPU= 13.588
  18:08:11 STEP 2561 H=2561:00 +CPU= 13.201
  18:08:32 STEP 2562 H=2562:00 +CPU= 21.470
  18:08:45 STEP 2563 H=2563:00 +CPU= 13.232
  18:08:59 STEP 2564 H=2564:00 +CPU= 13.593
  18:09:20 STEP 2565 H=2565:00 +CPU= 21.081
  18:09:34 STEP 2566 H=2566:00 +CPU= 13.560
  18:09:47 STEP 2567 H=2567:00 +CPU= 13.580
  18:10:11 STEP 2568 H=2568:00 +CPU= 24.278
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002568
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002568
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002568
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002544
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002544
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002544
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002556
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002556
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002556
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_106.zip
Uploading the intermediate file: upload_file_106.zip
  18:10:25 STEP 2569 H=2569:00 +CPU= 13.194
Uploading trickle at timestep: 9244800
  18:10:38 STEP 2570 H=2570:00 +CPU= 13.590
  18:10:59 STEP 2571 H=2571:00 +CPU= 20.990
  18:11:13 STEP 2572 H=2572:00 +CPU= 13.666
  18:11:26 STEP 2573 H=2573:00 +CPU= 13.230
  18:11:48 STEP 2574 H=2574:00 +CPU= 21.595
  18:12:01 STEP 2575 H=2575:00 +CPU= 13.182
  18:12:15 STEP 2576 H=2576:00 +CPU= 13.667
  18:12:36 STEP 2577 H=2577:00 +CPU= 21.142
  18:12:49 STEP 2578 H=2578:00 +CPU= 13.629
  18:13:03 STEP 2579 H=2579:00 +CPU= 13.191
  18:13:26 STEP 2580 H=2580:00 +CPU= 23.284
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002580
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002580
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002580
  18:13:39 STEP 2581 H=2581:00 +CPU= 13.256
  18:13:53 STEP 2582 H=2582:00 +CPU= 13.640
  18:14:14 STEP 2583 H=2583:00 +CPU= 21.144
  18:14:28 STEP 2584 H=2584:00 +CPU= 13.635
  18:14:41 STEP 2585 H=2585:00 +CPU= 13.235
  18:15:02 STEP 2586 H=2586:00 +CPU= 21.546
  18:15:16 STEP 2587 H=2587:00 +CPU= 13.235
  18:15:29 STEP 2588 H=2588:00 +CPU= 13.660
  18:15:50 STEP 2589 H=2589:00 +CPU= 21.113
  18:16:04 STEP 2590 H=2590:00 +CPU= 13.599
  18:16:18 STEP 2591 H=2591:00 +CPU= 13.602
  18:16:42 STEP 2592 H=2592:00 +CPU= 24.425
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002592
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002592
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002592
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002568
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002568
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002568
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002580
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002580
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002580
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_107.zip
Uploading the intermediate file: upload_file_107.zip
  18:16:55 STEP 2593 H=2593:00 +CPU= 13.255
Uploading trickle at timestep: 9331200
  18:17:09 STEP 2594 H=2594:00 +CPU= 13.632
  18:17:30 STEP 2595 H=2595:00 +CPU= 21.155
  18:17:44 STEP 2596 H=2596:00 +CPU= 13.675
  18:17:57 STEP 2597 H=2597:00 +CPU= 13.234
  18:18:18 STEP 2598 H=2598:00 +CPU= 21.588
  18:18:32 STEP 2599 H=2599:00 +CPU= 13.201
  18:18:45 STEP 2600 H=2600:00 +CPU= 13.674
  18:19:06 STEP 2601 H=2601:00 +CPU= 21.054
  18:19:20 STEP 2602 H=2602:00 +CPU= 13.663
  18:19:33 STEP 2603 H=2603:00 +CPU= 13.251
  18:19:57 STEP 2604 H=2604:00 +CPU= 23.343
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002604
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002604
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002604
  18:20:10 STEP 2605 H=2605:00 +CPU= 13.215
  18:20:23 STEP 2606 H=2606:00 +CPU= 13.608
  18:20:45 STEP 2607 H=2607:00 +CPU= 21.048
  18:20:58 STEP 2608 H=2608:00 +CPU= 13.601
  18:21:11 STEP 2609 H=2609:00 +CPU= 13.156
  18:21:33 STEP 2610 H=2610:00 +CPU= 21.542
  18:21:46 STEP 2611 H=2611:00 +CPU= 13.192
  18:22:00 STEP 2612 H=2612:00 +CPU= 13.598
  18:22:21 STEP 2613 H=2613:00 +CPU= 20.986
  18:22:34 STEP 2614 H=2614:00 +CPU= 13.585
  18:22:48 STEP 2615 H=2615:00 +CPU= 13.576
  18:23:12 STEP 2616 H=2616:00 +CPU= 24.222
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002616
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002616
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002616
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002592
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002592
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002592
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002604
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002604
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002604
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_108.zip
Uploading the intermediate file: upload_file_108.zip
  18:23:25 STEP 2617 H=2617:00 +CPU= 13.139
Uploading trickle at timestep: 9417600
  18:23:39 STEP 2618 H=2618:00 +CPU= 13.548
  18:24:00 STEP 2619 H=2619:00 +CPU= 20.982
  18:24:13 STEP 2620 H=2620:00 +CPU= 13.553
  18:24:26 STEP 2621 H=2621:00 +CPU= 13.187
  18:24:48 STEP 2622 H=2622:00 +CPU= 21.454
  18:25:01 STEP 2623 H=2623:00 +CPU= 13.228
  18:25:15 STEP 2624 H=2624:00 +CPU= 13.608
  18:25:36 STEP 2625 H=2625:00 +CPU= 21.011
  18:25:49 STEP 2626 H=2626:00 +CPU= 13.588
  18:26:03 STEP 2627 H=2627:00 +CPU= 13.194
  18:26:26 STEP 2628 H=2628:00 +CPU= 23.272
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002628
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002628
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002628
  18:26:39 STEP 2629 H=2629:00 +CPU= 13.215
  18:26:53 STEP 2630 H=2630:00 +CPU= 13.592
  18:27:14 STEP 2631 H=2631:00 +CPU= 20.981
  18:27:27 STEP 2632 H=2632:00 +CPU= 13.585
  18:27:40 STEP 2633 H=2633:00 +CPU= 13.161
  18:28:02 STEP 2634 H=2634:00 +CPU= 21.450
  18:28:15 STEP 2635 H=2635:00 +CPU= 13.186
  18:28:29 STEP 2636 H=2636:00 +CPU= 13.572
  18:28:50 STEP 2637 H=2637:00 +CPU= 21.014
  18:29:03 STEP 2638 H=2638:00 +CPU= 13.574
  18:29:17 STEP 2639 H=2639:00 +CPU= 13.543
  18:29:41 STEP 2640 H=2640:00 +CPU= 24.206
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002640
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002640
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002640
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002616
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002616
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002616
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002628
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002628
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002628
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_109.zip
Uploading the intermediate file: upload_file_109.zip
  18:29:54 STEP 2641 H=2641:00 +CPU= 13.176
Uploading trickle at timestep: 9504000
  18:30:08 STEP 2642 H=2642:00 +CPU= 13.562
  18:30:29 STEP 2643 H=2643:00 +CPU= 20.918
  18:30:42 STEP 2644 H=2644:00 +CPU= 13.544
  18:30:55 STEP 2645 H=2645:00 +CPU= 13.155
  18:31:17 STEP 2646 H=2646:00 +CPU= 21.585
  18:31:30 STEP 2647 H=2647:00 +CPU= 13.147
  18:31:44 STEP 2648 H=2648:00 +CPU= 13.567
  18:32:05 STEP 2649 H=2649:00 +CPU= 21.079
  18:32:18 STEP 2650 H=2650:00 +CPU= 13.566
  18:32:31 STEP 2651 H=2651:00 +CPU= 13.164
  18:32:55 STEP 2652 H=2652:00 +CPU= 23.254
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002652
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002652
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002652
  18:33:08 STEP 2653 H=2653:00 +CPU= 13.173
  18:33:21 STEP 2654 H=2654:00 +CPU= 13.590
  18:33:42 STEP 2655 H=2655:00 +CPU= 21.003
  18:33:56 STEP 2656 H=2656:00 +CPU= 13.614
  18:34:09 STEP 2657 H=2657:00 +CPU= 13.176
  18:34:31 STEP 2658 H=2658:00 +CPU= 21.520
  18:34:44 STEP 2659 H=2659:00 +CPU= 13.258
  18:34:58 STEP 2660 H=2660:00 +CPU= 13.593
  18:35:18 STEP 2661 H=2661:00 +CPU= 20.984
  18:35:32 STEP 2662 H=2662:00 +CPU= 13.578
  18:35:46 STEP 2663 H=2663:00 +CPU= 13.552
  18:36:10 STEP 2664 H=2664:00 +CPU= 24.358
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002664
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002664
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002664
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002640
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002640
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002640
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002652
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002652
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002652
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_110.zip
Uploading the intermediate file: upload_file_110.zip
  18:36:23 STEP 2665 H=2665:00 +CPU= 13.431
Uploading trickle at timestep: 9590400
  18:36:37 STEP 2666 H=2666:00 +CPU= 13.636
  18:36:58 STEP 2667 H=2667:00 +CPU= 21.069
  18:37:12 STEP 2668 H=2668:00 +CPU= 13.567
  18:37:25 STEP 2669 H=2669:00 +CPU= 13.287
  18:37:47 STEP 2670 H=2670:00 +CPU= 21.476
  18:38:00 STEP 2671 H=2671:00 +CPU= 13.183
  18:38:13 STEP 2672 H=2672:00 +CPU= 13.640
  18:38:34 STEP 2673 H=2673:00 +CPU= 21.008
  18:38:48 STEP 2674 H=2674:00 +CPU= 13.608
  18:39:01 STEP 2675 H=2675:00 +CPU= 13.183
  18:39:24 STEP 2676 H=2676:00 +CPU= 23.254
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002676
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002676
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002676
  18:39:38 STEP 2677 H=2677:00 +CPU= 13.203
  18:39:51 STEP 2678 H=2678:00 +CPU= 13.604
  18:40:12 STEP 2679 H=2679:00 +CPU= 21.095
  18:40:26 STEP 2680 H=2680:00 +CPU= 13.653
  18:40:39 STEP 2681 H=2681:00 +CPU= 13.190
  18:41:01 STEP 2682 H=2682:00 +CPU= 21.416
  18:41:14 STEP 2683 H=2683:00 +CPU= 13.184
  18:41:27 STEP 2684 H=2684:00 +CPU= 13.559
  18:41:48 STEP 2685 H=2685:00 +CPU= 21.015
  18:42:02 STEP 2686 H=2686:00 +CPU= 13.641
  18:42:16 STEP 2687 H=2687:00 +CPU= 13.639
  18:42:40 STEP 2688 H=2688:00 +CPU= 24.403
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002688
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002688
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002688
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002664
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002664
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002664
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002676
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002676
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002676
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_111.zip
Uploading the intermediate file: upload_file_111.zip
  18:42:53 STEP 2689 H=2689:00 +CPU= 13.203
  18:43:07 STEP 2690 H=2690:00 +CPU= 13.631
Uploading trickle at timestep: 9676800
  18:43:28 STEP 2691 H=2691:00 +CPU= 21.262
  18:43:42 STEP 2692 H=2692:00 +CPU= 13.642
  18:43:55 STEP 2693 H=2693:00 +CPU= 13.244
  18:44:17 STEP 2694 H=2694:00 +CPU= 21.768
  18:44:30 STEP 2695 H=2695:00 +CPU= 13.271
  18:44:44 STEP 2696 H=2696:00 +CPU= 13.641
  18:45:05 STEP 2697 H=2697:00 +CPU= 21.127
  18:45:18 STEP 2698 H=2698:00 +CPU= 13.612
  18:45:32 STEP 2699 H=2699:00 +CPU= 13.248
  18:45:55 STEP 2700 H=2700:00 +CPU= 23.391
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002700
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002700
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002700
  18:46:08 STEP 2701 H=2701:00 +CPU= 13.390
  18:46:22 STEP 2702 H=2702:00 +CPU= 13.736
  18:46:43 STEP 2703 H=2703:00 +CPU= 21.137
  18:46:57 STEP 2704 H=2704:00 +CPU= 13.678
  18:47:10 STEP 2705 H=2705:00 +CPU= 13.197
  18:47:32 STEP 2706 H=2706:00 +CPU= 21.624
  18:47:45 STEP 2707 H=2707:00 +CPU= 13.195
  18:47:59 STEP 2708 H=2708:00 +CPU= 13.573
  18:48:20 STEP 2709 H=2709:00 +CPU= 21.177
  18:48:33 STEP 2710 H=2710:00 +CPU= 13.698
  18:48:47 STEP 2711 H=2711:00 +CPU= 13.663
  18:49:11 STEP 2712 H=2712:00 +CPU= 24.369
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002712
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002712
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002712
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002688
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002688
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002688
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002700
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002700
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002700
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_112.zip
Uploading the intermediate file: upload_file_112.zip
  18:49:25 STEP 2713 H=2713:00 +CPU= 13.170
  18:49:38 STEP 2714 H=2714:00 +CPU= 13.618
Uploading trickle at timestep: 9763200
  18:49:59 STEP 2715 H=2715:00 +CPU= 21.207
  18:50:13 STEP 2716 H=2716:00 +CPU= 13.657
  18:50:26 STEP 2717 H=2717:00 +CPU= 13.197
  18:50:48 STEP 2718 H=2718:00 +CPU= 21.531
  18:51:01 STEP 2719 H=2719:00 +CPU= 13.263
  18:51:15 STEP 2720 H=2720:00 +CPU= 13.658
  18:51:36 STEP 2721 H=2721:00 +CPU= 21.097
  18:51:50 STEP 2722 H=2722:00 +CPU= 13.683
  18:52:03 STEP 2723 H=2723:00 +CPU= 13.254
  18:52:26 STEP 2724 H=2724:00 +CPU= 23.286
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002724
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002724
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002724
  18:52:39 STEP 2725 H=2725:00 +CPU= 13.252
  18:52:53 STEP 2726 H=2726:00 +CPU= 13.652
  18:53:14 STEP 2727 H=2727:00 +CPU= 21.087
  18:53:28 STEP 2728 H=2728:00 +CPU= 13.656
  18:53:41 STEP 2729 H=2729:00 +CPU= 13.234
  18:54:02 STEP 2730 H=2730:00 +CPU= 21.491
  18:54:16 STEP 2731 H=2731:00 +CPU= 13.213
  18:54:29 STEP 2732 H=2732:00 +CPU= 13.630
  18:54:50 STEP 2733 H=2733:00 +CPU= 21.012
  18:55:04 STEP 2734 H=2734:00 +CPU= 13.613
  18:55:18 STEP 2735 H=2735:00 +CPU= 13.612
  18:55:42 STEP 2736 H=2736:00 +CPU= 24.261
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002736
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002736
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002736
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002712
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002712
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002712
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002724
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002724
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002724
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_113.zip
Uploading the intermediate file: upload_file_113.zip
  18:55:55 STEP 2737 H=2737:00 +CPU= 13.260
  18:56:09 STEP 2738 H=2738:00 +CPU= 13.610
Uploading trickle at timestep: 9849600
  18:56:30 STEP 2739 H=2739:00 +CPU= 21.287
  18:56:44 STEP 2740 H=2740:00 +CPU= 13.628
  18:56:57 STEP 2741 H=2741:00 +CPU= 13.234
  18:57:18 STEP 2742 H=2742:00 +CPU= 21.536
  18:57:32 STEP 2743 H=2743:00 +CPU= 13.265
  18:57:45 STEP 2744 H=2744:00 +CPU= 13.654
  18:58:06 STEP 2745 H=2745:00 +CPU= 21.064
  18:58:20 STEP 2746 H=2746:00 +CPU= 13.688
  18:58:33 STEP 2747 H=2747:00 +CPU= 13.249
  18:58:57 STEP 2748 H=2748:00 +CPU= 23.325
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002748
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002748
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002748
  18:59:10 STEP 2749 H=2749:00 +CPU= 13.254
  18:59:24 STEP 2750 H=2750:00 +CPU= 13.667
  18:59:45 STEP 2751 H=2751:00 +CPU= 21.101
  18:59:58 STEP 2752 H=2752:00 +CPU= 13.637
  19:00:12 STEP 2753 H=2753:00 +CPU= 13.225
  19:00:33 STEP 2754 H=2754:00 +CPU= 21.567
  19:00:46 STEP 2755 H=2755:00 +CPU= 13.233
  19:01:00 STEP 2756 H=2756:00 +CPU= 13.621
  19:01:21 STEP 2757 H=2757:00 +CPU= 21.099
  19:01:35 STEP 2758 H=2758:00 +CPU= 13.639
  19:01:48 STEP 2759 H=2759:00 +CPU= 13.624
  19:02:13 STEP 2760 H=2760:00 +CPU= 24.296
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002760
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002760
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002760
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002736
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002736
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002736
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002748
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002748
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002748
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_114.zip
Uploading the intermediate file: upload_file_114.zip
  19:02:26 STEP 2761 H=2761:00 +CPU= 13.551
  19:02:40 STEP 2762 H=2762:00 +CPU= 13.663
Uploading trickle at timestep: 9936000
  19:03:01 STEP 2763 H=2763:00 +CPU= 21.529
  19:03:15 STEP 2764 H=2764:00 +CPU= 13.676
  19:03:28 STEP 2765 H=2765:00 +CPU= 13.219
  19:03:50 STEP 2766 H=2766:00 +CPU= 21.753
  19:04:03 STEP 2767 H=2767:00 +CPU= 13.363
  19:04:17 STEP 2768 H=2768:00 +CPU= 13.658
  19:04:38 STEP 2769 H=2769:00 +CPU= 21.117
  19:04:52 STEP 2770 H=2770:00 +CPU= 13.628
  19:05:05 STEP 2771 H=2771:00 +CPU= 13.249
  19:05:28 STEP 2772 H=2772:00 +CPU= 23.330
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002772
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002772
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002772
  19:05:42 STEP 2773 H=2773:00 +CPU= 13.372
  19:05:55 STEP 2774 H=2774:00 +CPU= 13.679
  19:06:17 STEP 2775 H=2775:00 +CPU= 21.247
  19:06:30 STEP 2776 H=2776:00 +CPU= 13.656
  19:06:44 STEP 2777 H=2777:00 +CPU= 13.239
  19:07:05 STEP 2778 H=2778:00 +CPU= 21.515
  19:07:18 STEP 2779 H=2779:00 +CPU= 13.281
  19:07:32 STEP 2780 H=2780:00 +CPU= 13.728
  19:07:53 STEP 2781 H=2781:00 +CPU= 21.155
  19:08:07 STEP 2782 H=2782:00 +CPU= 13.665
  19:08:21 STEP 2783 H=2783:00 +CPU= 13.637
  19:08:45 STEP 2784 H=2784:00 +CPU= 24.224
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002784
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002784
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002784
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002760
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002760
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002760
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002772
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002772
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002772
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_115.zip
Uploading the intermediate file: upload_file_115.zip
  19:08:58 STEP 2785 H=2785:00 +CPU= 13.258
  19:09:12 STEP 2786 H=2786:00 +CPU= 13.633
Uploading trickle at timestep: 10022400
  19:09:33 STEP 2787 H=2787:00 +CPU= 21.468
  19:09:47 STEP 2788 H=2788:00 +CPU= 13.594
  19:10:00 STEP 2789 H=2789:00 +CPU= 13.232
  19:10:22 STEP 2790 H=2790:00 +CPU= 21.604
  19:10:35 STEP 2791 H=2791:00 +CPU= 13.189
  19:10:48 STEP 2792 H=2792:00 +CPU= 13.679
  19:11:10 STEP 2793 H=2793:00 +CPU= 21.153
  19:11:23 STEP 2794 H=2794:00 +CPU= 13.677
  19:11:36 STEP 2795 H=2795:00 +CPU= 13.227
  19:12:00 STEP 2796 H=2796:00 +CPU= 23.374
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002796
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002796
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002796
  19:12:13 STEP 2797 H=2797:00 +CPU= 13.253
  19:12:27 STEP 2798 H=2798:00 +CPU= 13.682
  19:12:48 STEP 2799 H=2799:00 +CPU= 21.119
  19:13:02 STEP 2800 H=2800:00 +CPU= 13.683
  19:13:15 STEP 2801 H=2801:00 +CPU= 13.247
  19:13:36 STEP 2802 H=2802:00 +CPU= 21.659
  19:13:50 STEP 2803 H=2803:00 +CPU= 13.320
  19:14:03 STEP 2804 H=2804:00 +CPU= 13.686
  19:14:25 STEP 2805 H=2805:00 +CPU= 21.162
  19:14:38 STEP 2806 H=2806:00 +CPU= 13.680
  19:14:52 STEP 2807 H=2807:00 +CPU= 13.664
  19:15:16 STEP 2808 H=2808:00 +CPU= 24.255
Moving to projects directory: /home/ll/slots/3/ICMGGhpi1+002808
Moving to projects directory: /home/ll/slots/3/ICMSHhpi1+002808
Moving to projects directory: /home/ll/slots/3/ICMUAhpi1+002808
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002784
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002784
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002784
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMGGhpi1+002796
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMSHhpi1+002796
Adding to the zip: /home/ll/projects/climateprediction.net/oifs_43r3_ps_12165466/ICMUAhpi1+002796
Zipping up the intermediate file: /home/ll/projects/climateprediction.net/oifs_43r3_ps_2377_2021050100_123_947_12165466_1_r1598424611_116.zip
Uploading the intermediate file: upload_file_116.zip
  19:15:30 STEP 2809 H=2809:00 +CPU= 13.265
  19:15:43 STEP 2810 H=2810:00 +CPU= 13.614
Uploading trickle at timestep: 10108800
  19:16:04 STEP 2811 H=2811:00 +CPU= 21.223
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:23:09 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 10,108,800 45,294 0.0045
09 May 2023 15:23:09 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 10,022,400 44,903 0.0045
09 May 2023 15:23:09 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,936,000 44,511 0.0045
09 May 2023 15:23:09 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,849,600 44,120 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,763,200 43,728 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,676,800 43,337 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,590,400 42,945 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,504,000 42,554 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,417,600 42,162 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,331,200 41,777 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,244,800 41,386 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,158,400 40,994 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 9,072,000 40,603 0.0045
09 May 2023 15:22:56 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,985,600 40,212 0.0045
09 May 2023 15:22:43 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,899,200 39,830 0.0045
09 May 2023 15:22:43 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,812,800 39,439 0.0045
09 May 2023 15:22:43 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,726,400 39,048 0.0045
09 May 2023 15:22:43 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,640,000 38,667 0.0045
09 May 2023 15:22:43 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,553,600 38,306 0.0045
09 May 2023 15:22:43 1536643 22248579 oifs_43r3_ps_2377_2021050100_123_947_12165466_1 8,467,200 37,935 0.0045


©2024 climateprediction.net