climateprediction.net home page
Task 22295758

Task 22295758

Name oifs_43r3_ps_0796_2014050100_123_983_12200440_1
Workunit 12200440
Created 5 Jan 2023, 9:13:30 UTC
Sent 5 Jan 2023, 9:53:17 UTC
Report deadline 4 Feb 2023, 9:53:17 UTC
Received 6 Jan 2023, 5:36:12 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532338
Run time 16 hours 26 min 36 sec
CPU time 16 hours 24 min 58 sec
Validate state Invalid
Credit 11,975.04
Device peak FLOPS 5.62 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,793.12 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,628.94 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_91.zip
Uploading the intermediate file: upload_file_91.zip
  00:32:46 STEP 2209 H=2209:00 +CPU= 15.481
  00:33:02 STEP 2210 H=2210:00 +CPU= 16.019
  00:33:27 STEP 2211 H=2211:00 +CPU= 24.760
  00:33:42 STEP 2212 H=2212:00 +CPU= 15.302
  00:33:57 STEP 2213 H=2213:00 +CPU= 14.987
  00:34:23 STEP 2214 H=2214:00 +CPU= 25.278
  00:34:39 STEP 2215 H=2215:00 +CPU= 15.989
  00:34:54 STEP 2216 H=2216:00 +CPU= 15.618
  00:35:18 STEP 2217 H=2217:00 +CPU= 24.111
  00:35:34 STEP 2218 H=2218:00 +CPU= 15.756
  00:35:49 STEP 2219 H=2219:00 +CPU= 15.228
  00:36:16 STEP 2220 H=2220:00 +CPU= 27.182
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002220
  00:36:32 STEP 2221 H=2221:00 +CPU= 15.549
  00:36:48 STEP 2222 H=2222:00 +CPU= 15.689
  00:37:13 STEP 2223 H=2223:00 +CPU= 25.064
  00:37:28 STEP 2224 H=2224:00 +CPU= 14.995
  00:37:43 STEP 2225 H=2225:00 +CPU= 14.784
  00:38:08 STEP 2226 H=2226:00 +CPU= 25.265
  00:38:23 STEP 2227 H=2227:00 +CPU= 15.350
  00:38:39 STEP 2228 H=2228:00 +CPU= 15.665
  00:39:05 STEP 2229 H=2229:00 +CPU= 26.023
  00:39:21 STEP 2230 H=2230:00 +CPU= 16.608
  00:39:38 STEP 2231 H=2231:00 +CPU= 16.266
  00:40:09 STEP 2232 H=2232:00 +CPU= 31.373
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_92.zip
Uploading the intermediate file: upload_file_92.zip
  00:40:25 STEP 2233 H=2233:00 +CPU= 15.857
  00:40:41 STEP 2234 H=2234:00 +CPU= 16.290
  00:41:07 STEP 2235 H=2235:00 +CPU= 25.273
  00:41:23 STEP 2236 H=2236:00 +CPU= 16.051
  00:41:38 STEP 2237 H=2237:00 +CPU= 15.382
  00:42:03 STEP 2238 H=2238:00 +CPU= 24.782
  00:42:18 STEP 2239 H=2239:00 +CPU= 15.061
  00:42:34 STEP 2240 H=2240:00 +CPU= 16.278
  00:43:00 STEP 2241 H=2241:00 +CPU= 25.763
  00:43:17 STEP 2242 H=2242:00 +CPU= 17.421
  00:43:34 STEP 2243 H=2243:00 +CPU= 16.755
  00:44:03 STEP 2244 H=2244:00 +CPU= 28.706
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002244
  00:44:19 STEP 2245 H=2245:00 +CPU= 16.614
  00:44:36 STEP 2246 H=2246:00 +CPU= 16.776
  00:45:02 STEP 2247 H=2247:00 +CPU= 26.198
  00:45:19 STEP 2248 H=2248:00 +CPU= 16.540
  00:45:35 STEP 2249 H=2249:00 +CPU= 16.170
  00:46:02 STEP 2250 H=2250:00 +CPU= 26.455
  00:46:17 STEP 2251 H=2251:00 +CPU= 15.843
  00:46:34 STEP 2252 H=2252:00 +CPU= 16.386
  00:47:00 STEP 2253 H=2253:00 +CPU= 25.895
  00:47:16 STEP 2254 H=2254:00 +CPU= 16.788
  00:47:34 STEP 2255 H=2255:00 +CPU= 17.171
  00:48:04 STEP 2256 H=2256:00 +CPU= 30.570
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_93.zip
Uploading the intermediate file: upload_file_93.zip
  00:48:22 STEP 2257 H=2257:00 +CPU= 17.515
  00:48:38 STEP 2258 H=2258:00 +CPU= 15.965
  00:49:04 STEP 2259 H=2259:00 +CPU= 26.275
  00:49:20 STEP 2260 H=2260:00 +CPU= 15.916
  00:49:36 STEP 2261 H=2261:00 +CPU= 15.696
  00:50:02 STEP 2262 H=2262:00 +CPU= 26.545
  00:50:19 STEP 2263 H=2263:00 +CPU= 16.711
  00:50:36 STEP 2264 H=2264:00 +CPU= 16.810
  00:51:00 STEP 2265 H=2265:00 +CPU= 23.976
  00:51:15 STEP 2266 H=2266:00 +CPU= 15.546
  00:51:30 STEP 2267 H=2267:00 +CPU= 15.283
  00:51:58 STEP 2268 H=2268:00 +CPU= 27.625
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002268
  00:52:15 STEP 2269 H=2269:00 +CPU= 16.873
  00:52:32 STEP 2270 H=2270:00 +CPU= 17.251
  00:52:58 STEP 2271 H=2271:00 +CPU= 25.528
  00:53:13 STEP 2272 H=2272:00 +CPU= 15.743
  00:53:30 STEP 2273 H=2273:00 +CPU= 16.683
  00:53:57 STEP 2274 H=2274:00 +CPU= 26.592
  00:54:13 STEP 2275 H=2275:00 +CPU= 16.261
  00:54:30 STEP 2276 H=2276:00 +CPU= 16.893
  00:54:56 STEP 2277 H=2277:00 +CPU= 26.205
  00:55:14 STEP 2278 H=2278:00 +CPU= 17.870
  00:55:32 STEP 2279 H=2279:00 +CPU= 17.583
  00:56:03 STEP 2280 H=2280:00 +CPU= 31.429
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_94.zip
Uploading the intermediate file: upload_file_94.zip
  00:56:19 STEP 2281 H=2281:00 +CPU= 15.795
  00:56:36 STEP 2282 H=2282:00 +CPU= 16.932
  00:57:02 STEP 2283 H=2283:00 +CPU= 26.653
  00:57:19 STEP 2284 H=2284:00 +CPU= 16.386
  00:57:35 STEP 2285 H=2285:00 +CPU= 16.371
  00:58:01 STEP 2286 H=2286:00 +CPU= 26.119
  00:58:17 STEP 2287 H=2287:00 +CPU= 16.051
  00:58:33 STEP 2288 H=2288:00 +CPU= 15.430
  00:58:59 STEP 2289 H=2289:00 +CPU= 25.813
  00:59:15 STEP 2290 H=2290:00 +CPU= 16.156
  00:59:30 STEP 2291 H=2291:00 +CPU= 15.697
  00:59:58 STEP 2292 H=2292:00 +CPU= 27.425
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002292
  01:00:13 STEP 2293 H=2293:00 +CPU= 14.967
  01:00:28 STEP 2294 H=2294:00 +CPU= 15.583
  01:00:53 STEP 2295 H=2295:00 +CPU= 24.173
  01:01:08 STEP 2296 H=2296:00 +CPU= 15.613
  01:01:23 STEP 2297 H=2297:00 +CPU= 14.901
  01:01:48 STEP 2298 H=2298:00 +CPU= 25.003
  01:02:03 STEP 2299 H=2299:00 +CPU= 15.012
  01:02:19 STEP 2300 H=2300:00 +CPU= 15.546
  01:02:43 STEP 2301 H=2301:00 +CPU= 24.708
  01:02:59 STEP 2302 H=2302:00 +CPU= 15.408
  01:03:16 STEP 2303 H=2303:00 +CPU= 16.936
  01:03:47 STEP 2304 H=2304:00 +CPU= 31.688
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_95.zip
Uploading the intermediate file: upload_file_95.zip
  01:04:05 STEP 2305 H=2305:00 +CPU= 17.056
  01:04:22 STEP 2306 H=2306:00 +CPU= 17.938
  01:04:49 STEP 2307 H=2307:00 +CPU= 26.973
  01:05:07 STEP 2308 H=2308:00 +CPU= 17.939
  01:05:24 STEP 2309 H=2309:00 +CPU= 16.364
  01:05:51 STEP 2310 H=2310:00 +CPU= 27.281
  01:06:08 STEP 2311 H=2311:00 +CPU= 16.973
  01:06:25 STEP 2312 H=2312:00 +CPU= 17.093
  01:06:52 STEP 2313 H=2313:00 +CPU= 26.673
  01:07:09 STEP 2314 H=2314:00 +CPU= 17.112
  01:07:25 STEP 2315 H=2315:00 +CPU= 16.029
  01:07:55 STEP 2316 H=2316:00 +CPU= 30.394
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002316
  01:08:11 STEP 2317 H=2317:00 +CPU= 16.096
  01:08:28 STEP 2318 H=2318:00 +CPU= 16.242
  01:08:54 STEP 2319 H=2319:00 +CPU= 25.988
  01:09:10 STEP 2320 H=2320:00 +CPU= 16.210
  01:09:25 STEP 2321 H=2321:00 +CPU= 15.015
  01:09:52 STEP 2322 H=2322:00 +CPU= 27.274
  01:10:09 STEP 2323 H=2323:00 +CPU= 16.688
  01:10:25 STEP 2324 H=2324:00 +CPU= 16.253
  01:10:57 STEP 2325 H=2325:00 +CPU= 32.110
  01:11:20 STEP 2326 H=2326:00 +CPU= 22.821
  01:11:39 STEP 2327 H=2327:00 +CPU= 19.299
  01:12:17 STEP 2328 H=2328:00 +CPU= 38.021
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_96.zip
Uploading the intermediate file: upload_file_96.zip
  01:12:37 STEP 2329 H=2329:00 +CPU= 19.532
  01:13:00 STEP 2330 H=2330:00 +CPU= 22.857
  01:13:33 STEP 2331 H=2331:00 +CPU= 33.219
  01:13:49 STEP 2332 H=2332:00 +CPU= 15.533
  01:14:08 STEP 2333 H=2333:00 +CPU= 19.577
  01:14:45 STEP 2334 H=2334:00 +CPU= 36.369
  01:15:08 STEP 2335 H=2335:00 +CPU= 22.970
  01:15:32 STEP 2336 H=2336:00 +CPU= 24.005
  01:16:08 STEP 2337 H=2337:00 +CPU= 36.535
  01:16:31 STEP 2338 H=2338:00 +CPU= 23.204
  01:16:54 STEP 2339 H=2339:00 +CPU= 22.120
  01:17:33 STEP 2340 H=2340:00 +CPU= 39.128
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002340
  01:17:56 STEP 2341 H=2341:00 +CPU= 22.743
  01:18:20 STEP 2342 H=2342:00 +CPU= 24.493
  01:18:56 STEP 2343 H=2343:00 +CPU= 35.790
  01:19:19 STEP 2344 H=2344:00 +CPU= 23.467
  01:19:41 STEP 2345 H=2345:00 +CPU= 22.054
  01:20:19 STEP 2346 H=2346:00 +CPU= 37.448
  01:20:42 STEP 2347 H=2347:00 +CPU= 22.982
  01:21:04 STEP 2348 H=2348:00 +CPU= 22.409
  01:21:38 STEP 2349 H=2349:00 +CPU= 33.491
  01:21:59 STEP 2350 H=2350:00 +CPU= 21.747
  01:22:24 STEP 2351 H=2351:00 +CPU= 24.236
  01:23:04 STEP 2352 H=2352:00 +CPU= 40.814
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_97.zip
Uploading the intermediate file: upload_file_97.zip
  01:23:26 STEP 2353 H=2353:00 +CPU= 21.553
  01:23:49 STEP 2354 H=2354:00 +CPU= 22.632
  01:24:23 STEP 2355 H=2355:00 +CPU= 34.597
  01:24:46 STEP 2356 H=2356:00 +CPU= 22.453
  01:25:07 STEP 2357 H=2357:00 +CPU= 21.246
  01:25:43 STEP 2358 H=2358:00 +CPU= 35.712
  01:26:04 STEP 2359 H=2359:00 +CPU= 21.391
  01:26:26 STEP 2360 H=2360:00 +CPU= 21.853
  01:26:55 STEP 2361 H=2361:00 +CPU= 29.303
  01:27:13 STEP 2362 H=2362:00 +CPU= 17.576
  01:27:34 STEP 2363 H=2363:00 +CPU= 20.762
  01:28:16 STEP 2364 H=2364:00 +CPU= 41.912
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002364
  01:28:38 STEP 2365 H=2365:00 +CPU= 21.976
  01:29:00 STEP 2366 H=2366:00 +CPU= 22.578
  01:29:35 STEP 2367 H=2367:00 +CPU= 34.507
  01:29:56 STEP 2368 H=2368:00 +CPU= 21.400
  01:30:13 STEP 2369 H=2369:00 +CPU= 17.186
  01:30:49 STEP 2370 H=2370:00 +CPU= 36.047
  01:31:12 STEP 2371 H=2371:00 +CPU= 22.264
  01:31:34 STEP 2372 H=2372:00 +CPU= 22.468
  01:32:08 STEP 2373 H=2373:00 +CPU= 33.738
  01:32:26 STEP 2374 H=2374:00 +CPU= 18.473
  01:32:50 STEP 2375 H=2375:00 +CPU= 23.335
  01:33:30 STEP 2376 H=2376:00 +CPU= 39.856
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_98.zip
Uploading the intermediate file: upload_file_98.zip
  01:33:52 STEP 2377 H=2377:00 +CPU= 22.614
Uploading trickle at timestep: 8553600
  01:34:15 STEP 2378 H=2378:00 +CPU= 22.360
  01:34:46 STEP 2379 H=2379:00 +CPU= 31.735
  01:35:07 STEP 2380 H=2380:00 +CPU= 21.025
  01:35:28 STEP 2381 H=2381:00 +CPU= 20.595
  01:35:59 STEP 2382 H=2382:00 +CPU= 30.758
  01:36:19 STEP 2383 H=2383:00 +CPU= 20.512
  01:36:41 STEP 2384 H=2384:00 +CPU= 21.759
  01:37:14 STEP 2385 H=2385:00 +CPU= 33.128
  01:37:32 STEP 2386 H=2386:00 +CPU= 17.462
  01:37:51 STEP 2387 H=2387:00 +CPU= 19.785
  01:38:29 STEP 2388 H=2388:00 +CPU= 37.192
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002388
  01:38:50 STEP 2389 H=2389:00 +CPU= 21.504
  01:39:08 STEP 2390 H=2390:00 +CPU= 17.470
  01:39:35 STEP 2391 H=2391:00 +CPU= 27.247
  01:39:57 STEP 2392 H=2392:00 +CPU= 21.821
  01:40:17 STEP 2393 H=2393:00 +CPU= 20.213
  01:40:49 STEP 2394 H=2394:00 +CPU= 32.486
  01:41:08 STEP 2395 H=2395:00 +CPU= 18.306
  01:41:30 STEP 2396 H=2396:00 +CPU= 21.889
  01:42:03 STEP 2397 H=2397:00 +CPU= 33.747
  01:42:25 STEP 2398 H=2398:00 +CPU= 21.837
  01:42:47 STEP 2399 H=2399:00 +CPU= 21.507
  01:43:21 STEP 2400 H=2400:00 +CPU= 34.050
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_99.zip
Uploading the intermediate file: upload_file_99.zip
  01:43:41 STEP 2401 H=2401:00 +CPU= 20.278
  01:44:04 STEP 2402 H=2402:00 +CPU= 23.068
  01:44:36 STEP 2403 H=2403:00 +CPU= 32.261
  01:44:58 STEP 2404 H=2404:00 +CPU= 21.944
  01:45:14 STEP 2405 H=2405:00 +CPU= 15.706
  01:45:49 STEP 2406 H=2406:00 +CPU= 34.800
  01:46:11 STEP 2407 H=2407:00 +CPU= 22.617
  01:46:28 STEP 2408 H=2408:00 +CPU= 16.327
  01:46:55 STEP 2409 H=2409:00 +CPU= 27.239
  01:47:13 STEP 2410 H=2410:00 +CPU= 18.386
  01:47:30 STEP 2411 H=2411:00 +CPU= 16.554
  01:47:59 STEP 2412 H=2412:00 +CPU= 28.988
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002412
  01:48:15 STEP 2413 H=2413:00 +CPU= 16.244
  01:48:34 STEP 2414 H=2414:00 +CPU= 18.762
  01:49:02 STEP 2415 H=2415:00 +CPU= 28.307
  01:49:21 STEP 2416 H=2416:00 +CPU= 18.702
  01:49:39 STEP 2417 H=2417:00 +CPU= 17.569
  01:50:07 STEP 2418 H=2418:00 +CPU= 28.530
  01:50:23 STEP 2419 H=2419:00 +CPU= 15.948
  01:50:40 STEP 2420 H=2420:00 +CPU= 16.585
  01:51:06 STEP 2421 H=2421:00 +CPU= 26.673
  01:51:24 STEP 2422 H=2422:00 +CPU= 17.373
  01:51:41 STEP 2423 H=2423:00 +CPU= 17.307
  01:52:11 STEP 2424 H=2424:00 +CPU= 30.347
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_100.zip
Uploading the intermediate file: upload_file_100.zip
  01:52:28 STEP 2425 H=2425:00 +CPU= 17.134
  01:52:46 STEP 2426 H=2426:00 +CPU= 18.011
  01:53:11 STEP 2427 H=2427:00 +CPU= 24.733
  01:53:26 STEP 2428 H=2428:00 +CPU= 15.210
  01:53:43 STEP 2429 H=2429:00 +CPU= 16.676
  01:54:10 STEP 2430 H=2430:00 +CPU= 27.096
  01:54:27 STEP 2431 H=2431:00 +CPU= 16.934
  01:54:44 STEP 2432 H=2432:00 +CPU= 17.138
  01:55:11 STEP 2433 H=2433:00 +CPU= 26.397
  01:55:27 STEP 2434 H=2434:00 +CPU= 16.370
  01:55:44 STEP 2435 H=2435:00 +CPU= 16.691
  01:56:16 STEP 2436 H=2436:00 +CPU= 32.378
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002436
  01:56:34 STEP 2437 H=2437:00 +CPU= 17.436
  01:56:51 STEP 2438 H=2438:00 +CPU= 17.231
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            9  MAX ETADOT VERTICAL VEL.=   5.909318375173943E-007
  LON  =    152.400000000000       degrees
  LAT  =   -29.7194674653187       degrees
  01:57:17 STEP 2439 H=2439:00 +CPU= 26.547
  01:57:34 STEP 2440 H=2440:00 +CPU= 16.897
  01:57:51 STEP 2441 H=2441:00 +CPU= 16.451
  01:58:17 STEP 2442 H=2442:00 +CPU= 26.457
  01:58:34 STEP 2443 H=2443:00 +CPU= 16.542
  01:58:51 STEP 2444 H=2444:00 +CPU= 16.951
  01:59:17 STEP 2445 H=2445:00 +CPU= 26.260
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           28  MAX ETADOT VERTICAL VEL.=   3.100204830905957E-007
  LON  =    150.000000000000       degrees
  LAT  =   -33.0839310244466       degrees
  01:59:34 STEP 2446 H=2446:00 +CPU= 17.230
  01:59:52 STEP 2447 H=2447:00 +CPU= 17.633
  02:00:24 STEP 2448 H=2448:00 +CPU= 31.752
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_101.zip
Uploading the intermediate file: upload_file_101.zip
  02:00:41 STEP 2449 H=2449:00 +CPU= 16.951
  02:00:59 STEP 2450 H=2450:00 +CPU= 18.415
  02:01:26 STEP 2451 H=2451:00 +CPU= 26.842
  02:01:42 STEP 2452 H=2452:00 +CPU= 16.242
  02:01:58 STEP 2453 H=2453:00 +CPU= 16.332
  02:02:26 STEP 2454 H=2454:00 +CPU= 28.068
  02:02:43 STEP 2455 H=2455:00 +CPU= 16.957
  02:03:06 STEP 2456 H=2456:00 +CPU= 22.531
  02:03:42 STEP 2457 H=2457:00 +CPU= 35.815
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0796
(argv4) batchid: 983
(argv5) wuid: 12200440
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1c12d1b5ff49cc07846368d0e50d56da to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0796_2014050100_123_983_12200440.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0796_2014050100_123_983_12200440.zip
ic_ancil_file: ic_ancil_12200440
ifsdata_file: ifsdata_12200440
climate_data_file: clim_data_12200440
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dc06a2b1158e2b119892deb6b3eb7bd2 to: /var/lib/boinc-client/slots/21/ic_ancil_12200440.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12200440.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5b3995565b63ebce7f40700bd6ac8d21 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12200440.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12200440.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12200440.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12200440.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12200440.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12200440.xml
-- Model is restarting --
Adjusting last_iter, 2458, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12200440.xml
last_cpu_time: 52797
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685
The child process has been launched with process id: 588078
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588078:588078] [20230106:024101:1672972861:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:41:58 STEP 2448 H=2448:00 +CPU= 53.442
  02:42:28 STEP 2449 H=2449:00 +CPU= 29.835
  02:42:59 STEP 2450 H=2450:00 +CPU= 30.168
  02:43:44 STEP 2451 H=2451:00 +CPU= 45.743
  02:44:12 STEP 2452 H=2452:00 +CPU= 27.746
  02:44:39 STEP 2453 H=2453:00 +CPU= 27.379
  02:45:28 STEP 2454 H=2454:00 +CPU= 48.788
  02:45:57 STEP 2455 H=2455:00 +CPU= 29.056
  02:46:25 STEP 2456 H=2456:00 +CPU= 27.848
  02:47:15 STEP 2457 H=2457:00 +CPU= 49.957
  02:47:43 STEP 2458 H=2458:00 +CPU= 28.095
  02:48:10 STEP 2459 H=2459:00 +CPU= 26.774
  02:48:56 STEP 2460 H=2460:00 +CPU= 45.420
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002460
  02:49:21 STEP 2461 H=2461:00 +CPU= 25.321
  02:49:50 STEP 2462 H=2462:00 +CPU= 29.297
  02:50:46 STEP 2463 H=2463:00 +CPU= 55.812
  02:51:15 STEP 2464 H=2464:00 +CPU= 28.757
  02:51:45 STEP 2465 H=2465:00 +CPU= 29.732
  02:52:23 STEP 2466 H=2466:00 +CPU= 38.643
  02:52:53 STEP 2467 H=2467:00 +CPU= 29.553
  02:53:26 STEP 2468 H=2468:00 +CPU= 32.834
  02:54:08 STEP 2469 H=2469:00 +CPU= 42.627
  02:54:35 STEP 2470 H=2470:00 +CPU= 26.467
  02:54:55 STEP 2471 H=2471:00 +CPU= 20.034
  02:55:33 STEP 2472 H=2472:00 +CPU= 38.483
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_102.zip
Uploading the intermediate file: upload_file_102.zip
  02:55:56 STEP 2473 H=2473:00 +CPU= 22.365
  02:56:21 STEP 2474 H=2474:00 +CPU= 25.670
  02:56:59 STEP 2475 H=2475:00 +CPU= 37.668
  02:57:23 STEP 2476 H=2476:00 +CPU= 23.745
  02:57:44 STEP 2477 H=2477:00 +CPU= 21.483
  02:58:29 STEP 2478 H=2478:00 +CPU= 45.246
  02:58:48 STEP 2479 H=2479:00 +CPU= 18.659
  02:59:17 STEP 2480 H=2480:00 +CPU= 28.750
  02:59:54 STEP 2481 H=2481:00 +CPU= 36.814
  03:00:17 STEP 2482 H=2482:00 +CPU= 23.403
  03:00:38 STEP 2483 H=2483:00 +CPU= 21.283
  03:01:18 STEP 2484 H=2484:00 +CPU= 39.492
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002484
  03:01:43 STEP 2485 H=2485:00 +CPU= 25.034
  03:02:07 STEP 2486 H=2486:00 +CPU= 23.772
  03:02:53 STEP 2487 H=2487:00 +CPU= 45.622
  03:03:21 STEP 2488 H=2488:00 +CPU= 28.316
  03:03:48 STEP 2489 H=2489:00 +CPU= 26.376
  03:04:41 STEP 2490 H=2490:00 +CPU= 53.551
  03:05:09 STEP 2491 H=2491:00 +CPU= 27.332
  03:05:40 STEP 2492 H=2492:00 +CPU= 31.328
  03:06:26 STEP 2493 H=2493:00 +CPU= 45.777
  03:06:57 STEP 2494 H=2494:00 +CPU= 31.047
  03:07:30 STEP 2495 H=2495:00 +CPU= 32.879
  03:08:39 STEP 2496 H=2496:00 +CPU= 68.650
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_103.zip
Uploading the intermediate file: upload_file_103.zip
  03:09:22 STEP 2497 H=2497:00 +CPU= 42.878
  03:09:57 STEP 2498 H=2498:00 +CPU= 34.359
  03:10:53 STEP 2499 H=2499:00 +CPU= 56.206
  03:11:28 STEP 2500 H=2500:00 +CPU= 35.134
  03:11:55 STEP 2501 H=2501:00 +CPU= 26.871
  03:12:45 STEP 2502 H=2502:00 +CPU= 49.128
  03:13:17 STEP 2503 H=2503:00 +CPU= 32.272
  03:13:50 STEP 2504 H=2504:00 +CPU= 33.392
  03:14:42 STEP 2505 H=2505:00 +CPU= 51.681
  03:15:13 STEP 2506 H=2506:00 +CPU= 30.403
  03:15:41 STEP 2507 H=2507:00 +CPU= 28.470
  03:16:29 STEP 2508 H=2508:00 +CPU= 48.229
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002508
  03:16:54 STEP 2509 H=2509:00 +CPU= 24.789
  03:17:21 STEP 2510 H=2510:00 +CPU= 27.234
  03:18:05 STEP 2511 H=2511:00 +CPU= 43.476
  03:18:35 STEP 2512 H=2512:00 +CPU= 29.390
  03:19:02 STEP 2513 H=2513:00 +CPU= 27.642
  03:19:45 STEP 2514 H=2514:00 +CPU= 43.179
  03:20:12 STEP 2515 H=2515:00 +CPU= 26.366
  03:20:38 STEP 2516 H=2516:00 +CPU= 25.603
  03:21:26 STEP 2517 H=2517:00 +CPU= 48.025
  03:21:58 STEP 2518 H=2518:00 +CPU= 32.070
  03:22:34 STEP 2519 H=2519:00 +CPU= 35.312
  03:23:27 STEP 2520 H=2520:00 +CPU= 52.474
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_104.zip
Uploading the intermediate file: upload_file_104.zip
  03:23:54 STEP 2521 H=2521:00 +CPU= 27.031
  03:24:20 STEP 2522 H=2522:00 +CPU= 25.782
  03:25:00 STEP 2523 H=2523:00 +CPU= 39.860
  03:25:24 STEP 2524 H=2524:00 +CPU= 24.763
  03:25:48 STEP 2525 H=2525:00 +CPU= 23.902
  03:26:27 STEP 2526 H=2526:00 +CPU= 38.661
  03:26:49 STEP 2527 H=2527:00 +CPU= 22.069
  03:27:13 STEP 2528 H=2528:00 +CPU= 24.238
  03:27:50 STEP 2529 H=2529:00 +CPU= 36.960
  03:28:13 STEP 2530 H=2530:00 +CPU= 22.765
  03:28:36 STEP 2531 H=2531:00 +CPU= 22.661
  03:29:18 STEP 2532 H=2532:00 +CPU= 42.093
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002532
  03:29:42 STEP 2533 H=2533:00 +CPU= 23.971
  03:30:07 STEP 2534 H=2534:00 +CPU= 24.572
  03:30:44 STEP 2535 H=2535:00 +CPU= 36.643
  03:31:08 STEP 2536 H=2536:00 +CPU= 23.982
  03:31:31 STEP 2537 H=2537:00 +CPU= 22.986
  03:32:08 STEP 2538 H=2538:00 +CPU= 36.717
  03:32:29 STEP 2539 H=2539:00 +CPU= 21.429
  03:32:52 STEP 2540 H=2540:00 +CPU= 22.539
  03:33:27 STEP 2541 H=2541:00 +CPU= 35.435
  03:33:51 STEP 2542 H=2542:00 +CPU= 23.102
  03:34:16 STEP 2543 H=2543:00 +CPU= 25.767
  03:35:01 STEP 2544 H=2544:00 +CPU= 44.509
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_105.zip
Uploading the intermediate file: upload_file_105.zip
  03:35:24 STEP 2545 H=2545:00 +CPU= 23.288
  03:35:49 STEP 2546 H=2546:00 +CPU= 24.745
  03:36:29 STEP 2547 H=2547:00 +CPU= 39.461
  03:36:53 STEP 2548 H=2548:00 +CPU= 24.285
  03:37:16 STEP 2549 H=2549:00 +CPU= 22.841
  03:37:53 STEP 2550 H=2550:00 +CPU= 36.855
  03:38:16 STEP 2551 H=2551:00 +CPU= 22.808
  03:38:40 STEP 2552 H=2552:00 +CPU= 23.852
  03:39:20 STEP 2553 H=2553:00 +CPU= 40.147
  03:39:44 STEP 2554 H=2554:00 +CPU= 24.165
  03:40:07 STEP 2555 H=2555:00 +CPU= 22.853
  03:40:49 STEP 2556 H=2556:00 +CPU= 41.793
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002556
  03:41:11 STEP 2557 H=2557:00 +CPU= 22.770
  03:41:36 STEP 2558 H=2558:00 +CPU= 24.264
  03:42:16 STEP 2559 H=2559:00 +CPU= 40.124
  03:42:40 STEP 2560 H=2560:00 +CPU= 23.729
  03:43:00 STEP 2561 H=2561:00 +CPU= 19.894
  03:43:33 STEP 2562 H=2562:00 +CPU= 33.246
  03:43:55 STEP 2563 H=2563:00 +CPU= 22.051
  03:44:18 STEP 2564 H=2564:00 +CPU= 23.355
  03:44:56 STEP 2565 H=2565:00 +CPU= 37.680
  03:45:18 STEP 2566 H=2566:00 +CPU= 21.996
  03:45:41 STEP 2567 H=2567:00 +CPU= 23.092
  03:46:23 STEP 2568 H=2568:00 +CPU= 41.580
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_106.zip
Uploading the intermediate file: upload_file_106.zip
  03:46:42 STEP 2569 H=2569:00 +CPU= 19.302
  03:47:03 STEP 2570 H=2570:00 +CPU= 20.597
  03:47:39 STEP 2571 H=2571:00 +CPU= 36.845
  03:48:02 STEP 2572 H=2572:00 +CPU= 22.603
  03:48:21 STEP 2573 H=2573:00 +CPU= 18.795
  03:48:54 STEP 2574 H=2574:00 +CPU= 32.970
  03:49:14 STEP 2575 H=2575:00 +CPU= 20.128
  03:49:34 STEP 2576 H=2576:00 +CPU= 19.721
  03:50:07 STEP 2577 H=2577:00 +CPU= 33.636
  03:50:31 STEP 2578 H=2578:00 +CPU= 23.530
  03:50:52 STEP 2579 H=2579:00 +CPU= 21.554
  03:51:25 STEP 2580 H=2580:00 +CPU= 33.127
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002580
  03:51:47 STEP 2581 H=2581:00 +CPU= 21.928
  03:52:11 STEP 2582 H=2582:00 +CPU= 24.033
  03:52:49 STEP 2583 H=2583:00 +CPU= 37.700
  03:53:08 STEP 2584 H=2584:00 +CPU= 19.141
  03:53:30 STEP 2585 H=2585:00 +CPU= 21.607
  03:54:03 STEP 2586 H=2586:00 +CPU= 33.387
  03:54:24 STEP 2587 H=2587:00 +CPU= 20.298
  03:54:41 STEP 2588 H=2588:00 +CPU= 17.060
  03:55:08 STEP 2589 H=2589:00 +CPU= 27.006
  03:55:29 STEP 2590 H=2590:00 +CPU= 20.854
  03:55:51 STEP 2591 H=2591:00 +CPU= 22.152
  03:56:26 STEP 2592 H=2592:00 +CPU= 35.116
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_107.zip
Uploading the intermediate file: upload_file_107.zip
  03:56:44 STEP 2593 H=2593:00 +CPU= 18.244
  03:57:02 STEP 2594 H=2594:00 +CPU= 17.624
  03:57:31 STEP 2595 H=2595:00 +CPU= 29.747
  03:57:49 STEP 2596 H=2596:00 +CPU= 17.223
  03:58:06 STEP 2597 H=2597:00 +CPU= 17.328
  03:58:33 STEP 2598 H=2598:00 +CPU= 26.986
  03:58:53 STEP 2599 H=2599:00 +CPU= 20.162
  03:59:11 STEP 2600 H=2600:00 +CPU= 18.274
  03:59:44 STEP 2601 H=2601:00 +CPU= 32.318
  04:00:03 STEP 2602 H=2602:00 +CPU= 19.413
  04:00:18 STEP 2603 H=2603:00 +CPU= 14.911
  04:00:45 STEP 2604 H=2604:00 +CPU= 26.833
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002604
  04:01:00 STEP 2605 H=2605:00 +CPU= 15.155
  04:01:16 STEP 2606 H=2606:00 +CPU= 16.192
  04:01:42 STEP 2607 H=2607:00 +CPU= 25.659
  04:02:03 STEP 2608 H=2608:00 +CPU= 21.231
  04:02:22 STEP 2609 H=2609:00 +CPU= 18.752
  04:02:50 STEP 2610 H=2610:00 +CPU= 27.826
  04:03:07 STEP 2611 H=2611:00 +CPU= 17.654
  04:03:26 STEP 2612 H=2612:00 +CPU= 18.256
  04:03:55 STEP 2613 H=2613:00 +CPU= 29.497
  04:04:14 STEP 2614 H=2614:00 +CPU= 18.665
  04:04:32 STEP 2615 H=2615:00 +CPU= 18.108
  04:05:04 STEP 2616 H=2616:00 +CPU= 32.536
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_108.zip
Uploading the intermediate file: upload_file_108.zip
  04:05:22 STEP 2617 H=2617:00 +CPU= 17.610
  04:05:42 STEP 2618 H=2618:00 +CPU= 20.355
  04:06:16 STEP 2619 H=2619:00 +CPU= 33.399
  04:06:38 STEP 2620 H=2620:00 +CPU= 22.577
  04:07:00 STEP 2621 H=2621:00 +CPU= 21.155
  04:07:37 STEP 2622 H=2622:00 +CPU= 37.340
  04:08:01 STEP 2623 H=2623:00 +CPU= 23.554
  04:08:23 STEP 2624 H=2624:00 +CPU= 22.617
  04:08:54 STEP 2625 H=2625:00 +CPU= 31.235
  04:09:14 STEP 2626 H=2626:00 +CPU= 19.370
  04:09:34 STEP 2627 H=2627:00 +CPU= 20.698
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0796
(argv4) batchid: 983
(argv5) wuid: 12200440
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/21
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1c12d1b5ff49cc07846368d0e50d56da to: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0796_2014050100_123_983_12200440.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/21/oifs_43r3_ps_0796_2014050100_123_983_12200440.zip
ic_ancil_file: ic_ancil_12200440
ifsdata_file: ifsdata_12200440
climate_data_file: clim_data_12200440
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dc06a2b1158e2b119892deb6b3eb7bd2 to: /var/lib/boinc-client/slots/21/ic_ancil_12200440.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/21/ic_ancil_12200440.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5b3995565b63ebce7f40700bd6ac8d21 to: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12200440.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/21/ifsdata/ifsdata_12200440.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/21/159l_2/clim_data_12200440.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/21/159l_2/clim_data_12200440.zip
Checking for progress XML file: /var/lib/boinc-client/slots/21/progress_file_12200440.xml
Opened progress file ok : /var/lib/boinc-client/slots/21/progress_file_12200440.xml
-- Model is restarting --
Adjusting last_iter, 2628, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/21/progress_file_12200440.xml
last_cpu_time: 58118
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685
The child process has been launched with process id: 590759
Executing the command: /var/lib/boinc-client/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590759:590759] [20230106:041916:1672978756:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:20:07 STEP 2616 H=2616:00 +CPU= 47.856
  04:20:33 STEP 2617 H=2617:00 +CPU= 25.986
  04:21:08 STEP 2618 H=2618:00 +CPU= 34.213
  04:21:49 STEP 2619 H=2619:00 +CPU= 41.311
  04:22:15 STEP 2620 H=2620:00 +CPU= 25.760
  04:22:42 STEP 2621 H=2621:00 +CPU= 26.941
  04:23:27 STEP 2622 H=2622:00 +CPU= 45.156
  04:23:56 STEP 2623 H=2623:00 +CPU= 28.991
  04:24:28 STEP 2624 H=2624:00 +CPU= 32.070
  04:25:13 STEP 2625 H=2625:00 +CPU= 44.954
  04:25:39 STEP 2626 H=2626:00 +CPU= 26.257
  04:26:07 STEP 2627 H=2627:00 +CPU= 27.282
  04:26:54 STEP 2628 H=2628:00 +CPU= 47.250
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002628
  04:27:21 STEP 2629 H=2629:00 +CPU= 27.219
  04:27:50 STEP 2630 H=2630:00 +CPU= 28.573
  04:28:36 STEP 2631 H=2631:00 +CPU= 46.068
  04:29:02 STEP 2632 H=2632:00 +CPU= 26.289
  04:29:29 STEP 2633 H=2633:00 +CPU= 27.073
  04:30:11 STEP 2634 H=2634:00 +CPU= 41.341
  04:30:35 STEP 2635 H=2635:00 +CPU= 24.558
  04:30:59 STEP 2636 H=2636:00 +CPU= 24.016
  04:31:44 STEP 2637 H=2637:00 +CPU= 45.044
  04:32:10 STEP 2638 H=2638:00 +CPU= 25.327
  04:32:34 STEP 2639 H=2639:00 +CPU= 24.874
  04:33:22 STEP 2640 H=2640:00 +CPU= 47.237
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/21/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200440/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0796_2014050100_123_983_12200440_1_r903351685_109.zip
Uploading the intermediate file: upload_file_109.zip
  04:33:47 STEP 2641 H=2641:00 +CPU= 25.589
  04:34:16 STEP 2642 H=2642:00 +CPU= 28.737
  04:34:57 STEP 2643 H=2643:00 +CPU= 40.484
  04:35:23 STEP 2644 H=2644:00 +CPU= 26.419
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:46 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 8,553,600 50,992 0.0060
09 May 2023 16:09:43 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 7,689,600 45,928 0.0060
09 May 2023 16:09:43 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 7,689,600 45,928 0.0060
09 May 2023 16:09:39 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 6,825,600 40,814 0.0060
09 May 2023 16:09:11 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 4,320,000 21,788 0.0050
09 May 2023 16:09:07 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 3,456,000 17,706 0.0051
09 May 2023 16:09:04 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 2,592,000 12,933 0.0050
09 May 2023 16:09:00 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 1,728,000 8,600 0.0050
09 May 2023 16:09:00 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 1,728,000 8,600 0.0050
09 May 2023 16:08:57 1532338 22295758 oifs_43r3_ps_0796_2014050100_123_983_12200440_1 864,000 4,509 0.0052


©2024 climateprediction.net