climateprediction.net home page
Task 22271563

Task 22271563

Name oifs_43r3_ps_0556_2000050100_123_969_12186200_0
Workunit 12186200
Created 22 Dec 2022, 21:45:15 UTC
Sent 19 Jan 2023, 14:21:00 UTC
Report deadline 20 Mar 2023, 14:21:00 UTC
Received 20 Jan 2023, 9:18:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1537625
Run time 13 hours 49 min 39 sec
CPU time 13 hours 43 min 43 sec
Validate state Invalid
Credit 9,918.72
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,780.26 MB
Peak swap size 4,977.94 MB
Peak disk usage 2,550.77 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Process still present 5 min after writing finish file; aborting</message>
<stderr_txt>
0/ICMSHhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_94.zip
Uploading the intermediate file: upload_file_94.zip
  17:12:19 STEP 2281 H=2281:00 +CPU= 10.264
  17:12:30 STEP 2282 H=2282:00 +CPU= 10.393
  17:12:47 STEP 2283 H=2283:00 +CPU= 16.663
  17:12:57 STEP 2284 H=2284:00 +CPU= 10.636
  17:13:08 STEP 2285 H=2285:00 +CPU= 10.729
  17:13:25 STEP 2286 H=2286:00 +CPU= 16.603
  17:13:35 STEP 2287 H=2287:00 +CPU= 10.392
  17:13:45 STEP 2288 H=2288:00 +CPU= 10.480
  17:14:02 STEP 2289 H=2289:00 +CPU= 16.668
  17:14:13 STEP 2290 H=2290:00 +CPU= 10.668
  17:14:23 STEP 2291 H=2291:00 +CPU= 10.086
  17:14:41 STEP 2292 H=2292:00 +CPU= 18.202
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002292
  17:14:51 STEP 2293 H=2293:00 +CPU= 10.359
  17:15:02 STEP 2294 H=2294:00 +CPU= 10.492
  17:15:18 STEP 2295 H=2295:00 +CPU= 16.347
  17:15:29 STEP 2296 H=2296:00 +CPU= 10.634
  17:15:39 STEP 2297 H=2297:00 +CPU= 10.118
  17:15:56 STEP 2298 H=2298:00 +CPU= 16.799
  17:16:06 STEP 2299 H=2299:00 +CPU= 10.341
  17:16:17 STEP 2300 H=2300:00 +CPU= 10.605
  17:16:33 STEP 2301 H=2301:00 +CPU= 16.313
  17:16:44 STEP 2302 H=2302:00 +CPU= 10.581
  17:16:54 STEP 2303 H=2303:00 +CPU= 10.493
  17:17:13 STEP 2304 H=2304:00 +CPU= 19.127
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_95.zip
Uploading the intermediate file: upload_file_95.zip
  17:17:23 STEP 2305 H=2305:00 +CPU= 10.068
  17:17:34 STEP 2306 H=2306:00 +CPU= 10.644
  17:17:51 STEP 2307 H=2307:00 +CPU= 16.645
  17:18:01 STEP 2308 H=2308:00 +CPU= 10.396
  17:18:11 STEP 2309 H=2309:00 +CPU= 10.229
  17:18:28 STEP 2310 H=2310:00 +CPU= 16.811
  17:18:38 STEP 2311 H=2311:00 +CPU= 10.179
  17:18:49 STEP 2312 H=2312:00 +CPU= 10.626
  17:19:05 STEP 2313 H=2313:00 +CPU= 16.195
  17:19:16 STEP 2314 H=2314:00 +CPU= 10.729
  17:19:26 STEP 2315 H=2315:00 +CPU= 10.024
  17:19:44 STEP 2316 H=2316:00 +CPU= 17.751
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002316
  17:19:54 STEP 2317 H=2317:00 +CPU= 10.344
  17:20:04 STEP 2318 H=2318:00 +CPU= 10.579
  17:20:20 STEP 2319 H=2319:00 +CPU= 15.914
  17:20:31 STEP 2320 H=2320:00 +CPU= 10.804
  17:20:41 STEP 2321 H=2321:00 +CPU= 10.121
  17:20:58 STEP 2322 H=2322:00 +CPU= 16.253
  17:21:08 STEP 2323 H=2323:00 +CPU= 10.454
  17:21:18 STEP 2324 H=2324:00 +CPU= 10.307
  17:21:35 STEP 2325 H=2325:00 +CPU= 16.364
  17:21:45 STEP 2326 H=2326:00 +CPU= 10.687
  17:21:56 STEP 2327 H=2327:00 +CPU= 10.271
  17:22:14 STEP 2328 H=2328:00 +CPU= 18.664
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_96.zip
Uploading the intermediate file: upload_file_96.zip
  17:22:25 STEP 2329 H=2329:00 +CPU= 10.422
  17:22:35 STEP 2330 H=2330:00 +CPU= 10.275
  17:22:51 STEP 2331 H=2331:00 +CPU= 16.312
  17:23:02 STEP 2332 H=2332:00 +CPU= 10.639
  17:23:12 STEP 2333 H=2333:00 +CPU= 10.115
  17:23:29 STEP 2334 H=2334:00 +CPU= 16.503
  17:23:39 STEP 2335 H=2335:00 +CPU= 10.718
  17:23:49 STEP 2336 H=2336:00 +CPU= 10.129
  17:24:06 STEP 2337 H=2337:00 +CPU= 16.678
  17:24:16 STEP 2338 H=2338:00 +CPU= 10.330
  17:24:27 STEP 2339 H=2339:00 +CPU= 10.069
  17:24:45 STEP 2340 H=2340:00 +CPU= 17.979
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002340
  17:24:55 STEP 2341 H=2341:00 +CPU= 10.122
  17:25:05 STEP 2342 H=2342:00 +CPU= 10.348
  17:25:21 STEP 2343 H=2343:00 +CPU= 16.075
  17:25:32 STEP 2344 H=2344:00 +CPU= 10.673
  17:25:42 STEP 2345 H=2345:00 +CPU= 10.050
  17:25:58 STEP 2346 H=2346:00 +CPU= 16.495
  17:26:09 STEP 2347 H=2347:00 +CPU= 10.227
  17:26:19 STEP 2348 H=2348:00 +CPU= 10.336
  17:26:36 STEP 2349 H=2349:00 +CPU= 16.642
  17:26:46 STEP 2350 H=2350:00 +CPU= 10.418
  17:26:56 STEP 2351 H=2351:00 +CPU= 10.320
  17:27:16 STEP 2352 H=2352:00 +CPU= 19.290
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_97.zip
Uploading the intermediate file: upload_file_97.zip
  17:27:26 STEP 2353 H=2353:00 +CPU= 10.168
  17:27:37 STEP 2354 H=2354:00 +CPU= 10.928
  17:27:53 STEP 2355 H=2355:00 +CPU= 16.202
  17:28:03 STEP 2356 H=2356:00 +CPU= 10.370
  17:28:13 STEP 2357 H=2357:00 +CPU= 10.180
  17:28:30 STEP 2358 H=2358:00 +CPU= 16.968
  17:28:41 STEP 2359 H=2359:00 +CPU= 10.148
  17:28:51 STEP 2360 H=2360:00 +CPU= 10.584
  17:29:08 STEP 2361 H=2361:00 +CPU= 16.542
  17:29:18 STEP 2362 H=2362:00 +CPU= 10.552
  17:29:28 STEP 2363 H=2363:00 +CPU= 10.227
  17:29:47 STEP 2364 H=2364:00 +CPU= 18.366
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002364
  17:29:57 STEP 2365 H=2365:00 +CPU=  9.913
  17:30:07 STEP 2366 H=2366:00 +CPU= 10.316
  17:30:24 STEP 2367 H=2367:00 +CPU= 16.578
  17:30:34 STEP 2368 H=2368:00 +CPU= 10.370
  17:30:44 STEP 2369 H=2369:00 +CPU= 10.266
  17:31:01 STEP 2370 H=2370:00 +CPU= 16.762
  17:31:12 STEP 2371 H=2371:00 +CPU= 10.536
  17:31:22 STEP 2372 H=2372:00 +CPU= 10.490
  17:31:38 STEP 2373 H=2373:00 +CPU= 16.432
  17:31:50 STEP 2374 H=2374:00 +CPU= 11.102
  17:32:00 STEP 2375 H=2375:00 +CPU= 10.723
  17:32:19 STEP 2376 H=2376:00 +CPU= 18.912
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_98.zip
Uploading the intermediate file: upload_file_98.zip
  17:32:29 STEP 2377 H=2377:00 +CPU= 10.019
  17:32:40 STEP 2378 H=2378:00 +CPU= 10.427
  17:32:56 STEP 2379 H=2379:00 +CPU= 16.822
  17:33:07 STEP 2380 H=2380:00 +CPU= 10.271
  17:33:17 STEP 2381 H=2381:00 +CPU= 10.344
  17:33:34 STEP 2382 H=2382:00 +CPU= 16.777
  17:33:44 STEP 2383 H=2383:00 +CPU= 10.124
  17:33:54 STEP 2384 H=2384:00 +CPU= 10.427
  17:34:11 STEP 2385 H=2385:00 +CPU= 16.070
  17:34:22 STEP 2386 H=2386:00 +CPU= 11.103
  17:34:32 STEP 2387 H=2387:00 +CPU= 10.204
  17:34:50 STEP 2388 H=2388:00 +CPU= 18.298
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002388
  17:35:00 STEP 2389 H=2389:00 +CPU= 10.024
  17:35:10 STEP 2390 H=2390:00 +CPU= 10.292
  17:35:27 STEP 2391 H=2391:00 +CPU= 16.736
  17:35:38 STEP 2392 H=2392:00 +CPU= 10.480
  17:35:48 STEP 2393 H=2393:00 +CPU= 10.294
  17:36:04 STEP 2394 H=2394:00 +CPU= 16.514
  17:36:14 STEP 2395 H=2395:00 +CPU=  9.976
  17:36:25 STEP 2396 H=2396:00 +CPU= 10.331
  17:36:41 STEP 2397 H=2397:00 +CPU= 16.313
  17:36:52 STEP 2398 H=2398:00 +CPU= 10.819
  17:37:02 STEP 2399 H=2399:00 +CPU= 10.334
  17:37:21 STEP 2400 H=2400:00 +CPU= 19.072
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_99.zip
Uploading the intermediate file: upload_file_99.zip
  17:37:31 STEP 2401 H=2401:00 +CPU=  9.856
  17:37:41 STEP 2402 H=2402:00 +CPU= 10.299
  17:37:58 STEP 2403 H=2403:00 +CPU= 16.510
  17:38:08 STEP 2404 H=2404:00 +CPU= 10.146
  17:38:19 STEP 2405 H=2405:00 +CPU= 10.455
  17:38:35 STEP 2406 H=2406:00 +CPU= 16.648
  17:38:45 STEP 2407 H=2407:00 +CPU= 10.196
  17:38:56 STEP 2408 H=2408:00 +CPU= 10.431
  17:39:13 STEP 2409 H=2409:00 +CPU= 16.680
  17:39:23 STEP 2410 H=2410:00 +CPU= 10.505
  17:39:33 STEP 2411 H=2411:00 +CPU= 10.005
  17:39:51 STEP 2412 H=2412:00 +CPU= 18.404
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002412
  17:40:01 STEP 2413 H=2413:00 +CPU=  9.874
  17:40:12 STEP 2414 H=2414:00 +CPU= 10.189
  17:40:28 STEP 2415 H=2415:00 +CPU= 16.565
  17:40:38 STEP 2416 H=2416:00 +CPU= 10.290
  17:40:49 STEP 2417 H=2417:00 +CPU= 10.326
  17:41:06 STEP 2418 H=2418:00 +CPU= 16.831
  17:41:16 STEP 2419 H=2419:00 +CPU=  9.967
  17:41:26 STEP 2420 H=2420:00 +CPU= 10.435
  17:41:42 STEP 2421 H=2421:00 +CPU= 16.300
  17:41:53 STEP 2422 H=2422:00 +CPU= 10.431
  17:42:03 STEP 2423 H=2423:00 +CPU= 10.507
  17:42:22 STEP 2424 H=2424:00 +CPU= 19.215
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_100.zip
Uploading the intermediate file: upload_file_100.zip
  17:42:33 STEP 2425 H=2425:00 +CPU= 10.104
  17:42:43 STEP 2426 H=2426:00 +CPU= 10.321
  17:42:59 STEP 2427 H=2427:00 +CPU= 16.487
  17:43:10 STEP 2428 H=2428:00 +CPU= 10.630
  17:43:20 STEP 2429 H=2429:00 +CPU= 10.242
  17:43:37 STEP 2430 H=2430:00 +CPU= 16.550
  17:43:47 STEP 2431 H=2431:00 +CPU= 10.158
  17:43:57 STEP 2432 H=2432:00 +CPU= 10.513
  17:44:14 STEP 2433 H=2433:00 +CPU= 16.748
  17:44:25 STEP 2434 H=2434:00 +CPU= 10.321
  17:44:35 STEP 2435 H=2435:00 +CPU= 10.658
  17:44:53 STEP 2436 H=2436:00 +CPU= 18.066
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002436
  17:45:03 STEP 2437 H=2437:00 +CPU=  9.938
  17:45:14 STEP 2438 H=2438:00 +CPU= 10.403
  17:45:30 STEP 2439 H=2439:00 +CPU= 16.264
  17:45:41 STEP 2440 H=2440:00 +CPU= 10.657
  17:45:51 STEP 2441 H=2441:00 +CPU= 10.130
  17:46:07 STEP 2442 H=2442:00 +CPU= 16.501
  17:46:18 STEP 2443 H=2443:00 +CPU= 10.975
  17:46:29 STEP 2444 H=2444:00 +CPU= 10.484
  17:46:45 STEP 2445 H=2445:00 +CPU= 16.157
  17:46:55 STEP 2446 H=2446:00 +CPU= 10.549
  17:47:06 STEP 2447 H=2447:00 +CPU= 10.369
  17:47:25 STEP 2448 H=2448:00 +CPU= 19.480
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_101.zip
Uploading the intermediate file: upload_file_101.zip
  17:47:35 STEP 2449 H=2449:00 +CPU= 10.115
  17:47:46 STEP 2450 H=2450:00 +CPU= 10.599
Uploading trickle at timestep: 8812800
  17:48:03 STEP 2451 H=2451:00 +CPU= 16.585
  17:48:13 STEP 2452 H=2452:00 +CPU= 10.179
  17:48:23 STEP 2453 H=2453:00 +CPU= 10.271
  17:48:40 STEP 2454 H=2454:00 +CPU= 16.633
  17:48:50 STEP 2455 H=2455:00 +CPU= 10.607
  17:49:01 STEP 2456 H=2456:00 +CPU= 10.467
  17:49:17 STEP 2457 H=2457:00 +CPU= 16.172
  17:49:27 STEP 2458 H=2458:00 +CPU= 10.366
  17:49:37 STEP 2459 H=2459:00 +CPU= 10.088
  17:49:56 STEP 2460 H=2460:00 +CPU= 18.389
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002460
  17:50:06 STEP 2461 H=2461:00 +CPU= 10.201
  17:50:16 STEP 2462 H=2462:00 +CPU= 10.487
  17:50:33 STEP 2463 H=2463:00 +CPU= 16.334
  17:50:43 STEP 2464 H=2464:00 +CPU= 10.293
  17:50:53 STEP 2465 H=2465:00 +CPU= 10.214
  17:51:10 STEP 2466 H=2466:00 +CPU= 16.699
  17:51:20 STEP 2467 H=2467:00 +CPU=  9.994
  17:51:31 STEP 2468 H=2468:00 +CPU= 10.947
  17:51:47 STEP 2469 H=2469:00 +CPU= 16.374
  17:51:58 STEP 2470 H=2470:00 +CPU= 10.915
  17:52:09 STEP 2471 H=2471:00 +CPU= 10.400
  17:52:28 STEP 2472 H=2472:00 +CPU= 19.025
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_102.zip
Uploading the intermediate file: upload_file_102.zip
  17:52:38 STEP 2473 H=2473:00 +CPU= 10.189
  17:52:48 STEP 2474 H=2474:00 +CPU= 10.383
  17:53:05 STEP 2475 H=2475:00 +CPU= 16.547
  17:53:15 STEP 2476 H=2476:00 +CPU= 10.320
  17:53:25 STEP 2477 H=2477:00 +CPU=  9.992
  17:53:42 STEP 2478 H=2478:00 +CPU= 16.765
  17:53:52 STEP 2479 H=2479:00 +CPU= 10.052
  17:54:03 STEP 2480 H=2480:00 +CPU= 10.792
  17:54:19 STEP 2481 H=2481:00 +CPU= 16.420
  17:54:30 STEP 2482 H=2482:00 +CPU= 11.158
  17:54:40 STEP 2483 H=2483:00 +CPU= 10.192
  17:54:58 STEP 2484 H=2484:00 +CPU= 18.005
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002484
  17:55:09 STEP 2485 H=2485:00 +CPU= 10.151
  17:55:19 STEP 2486 H=2486:00 +CPU= 10.192
  17:55:36 STEP 2487 H=2487:00 +CPU= 16.677
  17:55:46 STEP 2488 H=2488:00 +CPU= 10.012
  17:55:56 STEP 2489 H=2489:00 +CPU= 10.270
  17:56:12 STEP 2490 H=2490:00 +CPU= 16.667
  17:56:23 STEP 2491 H=2491:00 +CPU= 10.194
  17:56:33 STEP 2492 H=2492:00 +CPU= 10.220
  17:56:49 STEP 2493 H=2493:00 +CPU= 16.589
  17:57:00 STEP 2494 H=2494:00 +CPU= 10.973
  17:57:11 STEP 2495 H=2495:00 +CPU= 10.330
  17:57:30 STEP 2496 H=2496:00 +CPU= 19.066
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_103.zip
Uploading the intermediate file: upload_file_103.zip
  17:57:40 STEP 2497 H=2497:00 +CPU= 10.319
  17:57:50 STEP 2498 H=2498:00 +CPU= 10.319
  17:58:07 STEP 2499 H=2499:00 +CPU= 16.635
  17:58:18 STEP 2500 H=2500:00 +CPU= 10.897
  17:58:28 STEP 2501 H=2501:00 +CPU=  9.996
  17:58:45 STEP 2502 H=2502:00 +CPU= 17.259
  17:58:56 STEP 2503 H=2503:00 +CPU= 10.333
  17:59:06 STEP 2504 H=2504:00 +CPU= 10.647
  17:59:23 STEP 2505 H=2505:00 +CPU= 16.663
  17:59:34 STEP 2506 H=2506:00 +CPU= 10.651
  17:59:44 STEP 2507 H=2507:00 +CPU= 10.142
  18:00:02 STEP 2508 H=2508:00 +CPU= 18.142
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002508
  18:00:12 STEP 2509 H=2509:00 +CPU= 10.355
  18:00:23 STEP 2510 H=2510:00 +CPU= 10.596
  18:00:39 STEP 2511 H=2511:00 +CPU= 16.314
  18:00:50 STEP 2512 H=2512:00 +CPU= 10.442
  18:01:00 STEP 2513 H=2513:00 +CPU= 10.123
  18:01:17 STEP 2514 H=2514:00 +CPU= 16.958
  18:01:27 STEP 2515 H=2515:00 +CPU= 10.000
  18:01:37 STEP 2516 H=2516:00 +CPU= 10.429
  18:01:54 STEP 2517 H=2517:00 +CPU= 16.434
  18:02:04 STEP 2518 H=2518:00 +CPU= 10.299
  18:02:14 STEP 2519 H=2519:00 +CPU= 10.540
  18:02:33 STEP 2520 H=2520:00 +CPU= 18.916
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_104.zip
Uploading the intermediate file: upload_file_104.zip
  18:02:44 STEP 2521 H=2521:00 +CPU= 10.278
  18:02:54 STEP 2522 H=2522:00 +CPU= 10.366
  18:03:11 STEP 2523 H=2523:00 +CPU= 16.683
  18:03:22 STEP 2524 H=2524:00 +CPU= 10.960
  18:03:32 STEP 2525 H=2525:00 +CPU= 10.182
  18:03:48 STEP 2526 H=2526:00 +CPU= 16.725
  18:03:59 STEP 2527 H=2527:00 +CPU= 10.083
  18:04:09 STEP 2528 H=2528:00 +CPU= 10.283
  18:04:25 STEP 2529 H=2529:00 +CPU= 16.624
  18:04:36 STEP 2530 H=2530:00 +CPU= 10.412
  18:04:46 STEP 2531 H=2531:00 +CPU= 10.091
  18:05:04 STEP 2532 H=2532:00 +CPU= 18.295
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002532
  18:05:14 STEP 2533 H=2533:00 +CPU= 10.155
  18:05:25 STEP 2534 H=2534:00 +CPU= 10.543
  18:05:41 STEP 2535 H=2535:00 +CPU= 16.389
  18:05:52 STEP 2536 H=2536:00 +CPU= 10.397
  18:06:02 STEP 2537 H=2537:00 +CPU= 10.012
  18:06:18 STEP 2538 H=2538:00 +CPU= 16.555
  18:06:29 STEP 2539 H=2539:00 +CPU= 10.755
  18:06:39 STEP 2540 H=2540:00 +CPU= 10.254
  18:06:56 STEP 2541 H=2541:00 +CPU= 16.284
  18:07:06 STEP 2542 H=2542:00 +CPU= 10.655
  18:07:17 STEP 2543 H=2543:00 +CPU= 10.529
  18:07:36 STEP 2544 H=2544:00 +CPU= 18.972
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_105.zip
Uploading the intermediate file: upload_file_105.zip
  18:07:46 STEP 2545 H=2545:00 +CPU= 10.250
  18:07:57 STEP 2546 H=2546:00 +CPU= 10.618
  18:08:13 STEP 2547 H=2547:00 +CPU= 16.373
  18:08:23 STEP 2548 H=2548:00 +CPU= 10.251
  18:08:34 STEP 2549 H=2549:00 +CPU= 10.099
  18:08:50 STEP 2550 H=2550:00 +CPU= 16.517
  18:09:01 STEP 2551 H=2551:00 +CPU= 10.629
  18:09:11 STEP 2552 H=2552:00 +CPU= 10.318
  18:09:27 STEP 2553 H=2553:00 +CPU= 16.355
  18:09:38 STEP 2554 H=2554:00 +CPU= 10.631
  18:09:48 STEP 2555 H=2555:00 +CPU= 10.225
  18:10:07 STEP 2556 H=2556:00 +CPU= 18.631
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002556
  18:10:17 STEP 2557 H=2557:00 +CPU= 10.194
  18:10:27 STEP 2558 H=2558:00 +CPU= 10.254
  18:10:44 STEP 2559 H=2559:00 +CPU= 16.467
  18:10:54 STEP 2560 H=2560:00 +CPU= 10.475
  18:11:05 STEP 2561 H=2561:00 +CPU= 10.307
  18:11:21 STEP 2562 H=2562:00 +CPU= 16.642
  18:11:31 STEP 2563 H=2563:00 +CPU= 10.242
  18:11:42 STEP 2564 H=2564:00 +CPU= 10.408
  18:11:58 STEP 2565 H=2565:00 +CPU= 15.935
  18:12:09 STEP 2566 H=2566:00 +CPU= 10.971
  18:12:19 STEP 2567 H=2567:00 +CPU= 10.451
  18:12:39 STEP 2568 H=2568:00 +CPU= 19.341
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_106.zip
Uploading the intermediate file: upload_file_106.zip
  18:12:49 STEP 2569 H=2569:00 +CPU= 10.188
  18:12:59 STEP 2570 H=2570:00 +CPU= 10.270
  18:13:16 STEP 2571 H=2571:00 +CPU= 16.456
  18:13:26 STEP 2572 H=2572:00 +CPU= 10.358
  18:13:36 STEP 2573 H=2573:00 +CPU= 10.352
  18:13:53 STEP 2574 H=2574:00 +CPU= 16.789
  18:14:03 STEP 2575 H=2575:00 +CPU= 10.333
  18:14:14 STEP 2576 H=2576:00 +CPU= 10.546
  18:14:30 STEP 2577 H=2577:00 +CPU= 16.304
  18:14:41 STEP 2578 H=2578:00 +CPU= 10.489
  18:14:51 STEP 2579 H=2579:00 +CPU= 10.176
  18:15:09 STEP 2580 H=2580:00 +CPU= 18.226
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002580
  18:15:19 STEP 2581 H=2581:00 +CPU= 10.243
  18:15:30 STEP 2582 H=2582:00 +CPU= 10.344
  18:15:47 STEP 2583 H=2583:00 +CPU= 16.897
  18:15:57 STEP 2584 H=2584:00 +CPU= 10.304
  18:16:07 STEP 2585 H=2585:00 +CPU= 10.179
  18:16:24 STEP 2586 H=2586:00 +CPU= 16.689
  18:16:34 STEP 2587 H=2587:00 +CPU= 10.273
  18:16:45 STEP 2588 H=2588:00 +CPU= 10.641
  18:17:01 STEP 2589 H=2589:00 +CPU= 16.510
  18:17:12 STEP 2590 H=2590:00 +CPU= 10.576
  18:17:22 STEP 2591 H=2591:00 +CPU= 10.301
  18:17:41 STEP 2592 H=2592:00 +CPU= 19.170
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_107.zip
Uploading the intermediate file: upload_file_107.zip
  18:17:52 STEP 2593 H=2593:00 +CPU= 10.558
  18:18:02 STEP 2594 H=2594:00 +CPU= 10.272
  18:18:19 STEP 2595 H=2595:00 +CPU= 16.879
  18:18:29 STEP 2596 H=2596:00 +CPU= 10.236
  18:18:39 STEP 2597 H=2597:00 +CPU=  9.949
  18:18:56 STEP 2598 H=2598:00 +CPU= 16.833
  18:19:06 STEP 2599 H=2599:00 +CPU=  9.984
  18:19:17 STEP 2600 H=2600:00 +CPU= 10.658
  18:19:33 STEP 2601 H=2601:00 +CPU= 16.631
  18:19:44 STEP 2602 H=2602:00 +CPU= 10.402
  18:19:54 STEP 2603 H=2603:00 +CPU= 10.319
  18:20:12 STEP 2604 H=2604:00 +CPU= 18.199
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002604
  18:20:23 STEP 2605 H=2605:00 +CPU= 10.546
  18:20:33 STEP 2606 H=2606:00 +CPU= 10.389
  18:20:49 STEP 2607 H=2607:00 +CPU= 16.356
  18:21:00 STEP 2608 H=2608:00 +CPU= 10.789
  18:21:10 STEP 2609 H=2609:00 +CPU= 10.066
  18:21:28 STEP 2610 H=2610:00 +CPU= 17.200
  18:21:38 STEP 2611 H=2611:00 +CPU= 10.027
  18:21:48 STEP 2612 H=2612:00 +CPU= 10.538
  18:22:05 STEP 2613 H=2613:00 +CPU= 16.420
  18:22:15 STEP 2614 H=2614:00 +CPU= 10.182
  18:22:25 STEP 2615 H=2615:00 +CPU= 10.711
  18:22:44 STEP 2616 H=2616:00 +CPU= 18.973
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_108.zip
Uploading the intermediate file: upload_file_108.zip
  18:22:55 STEP 2617 H=2617:00 +CPU= 10.141
  18:23:05 STEP 2618 H=2618:00 +CPU= 10.393
  18:23:21 STEP 2619 H=2619:00 +CPU= 16.232
  18:23:32 STEP 2620 H=2620:00 +CPU= 10.641
  18:23:42 STEP 2621 H=2621:00 +CPU=  9.937
  18:23:59 STEP 2622 H=2622:00 +CPU= 17.151
  18:24:09 STEP 2623 H=2623:00 +CPU=  9.934
  18:24:19 STEP 2624 H=2624:00 +CPU= 10.446
  18:24:36 STEP 2625 H=2625:00 +CPU= 16.650
  18:24:46 STEP 2626 H=2626:00 +CPU= 10.455
  18:24:56 STEP 2627 H=2627:00 +CPU= 10.112
  18:25:15 STEP 2628 H=2628:00 +CPU= 18.093
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002628
  18:25:25 STEP 2629 H=2629:00 +CPU= 10.062
  18:25:35 STEP 2630 H=2630:00 +CPU= 10.576
  18:25:52 STEP 2631 H=2631:00 +CPU= 16.492
  18:26:02 STEP 2632 H=2632:00 +CPU= 10.518
  18:26:12 STEP 2633 H=2633:00 +CPU=  9.992
  18:26:29 STEP 2634 H=2634:00 +CPU= 16.686
  18:26:39 STEP 2635 H=2635:00 +CPU= 10.438
  18:26:50 STEP 2636 H=2636:00 +CPU= 10.209
  18:27:06 STEP 2637 H=2637:00 +CPU= 16.831
  18:27:17 STEP 2638 H=2638:00 +CPU= 10.340
  18:27:27 STEP 2639 H=2639:00 +CPU= 10.380
  18:27:46 STEP 2640 H=2640:00 +CPU= 18.956
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_109.zip
Uploading the intermediate file: upload_file_109.zip
  18:27:56 STEP 2641 H=2641:00 +CPU= 10.048
  18:28:07 STEP 2642 H=2642:00 +CPU= 10.440
  18:28:23 STEP 2643 H=2643:00 +CPU= 16.429
  18:28:34 STEP 2644 H=2644:00 +CPU= 11.010
  18:28:44 STEP 2645 H=2645:00 +CPU= 10.152
  18:29:01 STEP 2646 H=2646:00 +CPU= 17.089
  18:29:11 STEP 2647 H=2647:00 +CPU= 10.210
  18:29:22 STEP 2648 H=2648:00 +CPU= 10.555
  18:29:39 STEP 2649 H=2649:00 +CPU= 16.792
  18:29:49 STEP 2650 H=2650:00 +CPU= 10.376
  18:30:00 STEP 2651 H=2651:00 +CPU= 10.324
  18:30:17 STEP 2652 H=2652:00 +CPU= 17.921
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002652
  18:30:28 STEP 2653 H=2653:00 +CPU= 10.271
  18:30:38 STEP 2654 H=2654:00 +CPU= 10.422
  18:30:55 STEP 2655 H=2655:00 +CPU= 16.530
  18:31:06 STEP 2656 H=2656:00 +CPU= 11.085
  18:31:16 STEP 2657 H=2657:00 +CPU=  9.960
  18:31:32 STEP 2658 H=2658:00 +CPU= 16.541
  18:31:42 STEP 2659 H=2659:00 +CPU= 10.153
  18:31:53 STEP 2660 H=2660:00 +CPU= 10.459
  18:32:09 STEP 2661 H=2661:00 +CPU= 16.308
  18:32:20 STEP 2662 H=2662:00 +CPU= 10.405
  18:32:30 STEP 2663 H=2663:00 +CPU= 10.083
  18:32:49 STEP 2664 H=2664:00 +CPU= 19.013
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_110.zip
Uploading the intermediate file: upload_file_110.zip
  18:32:59 STEP 2665 H=2665:00 +CPU= 10.343
  18:33:09 STEP 2666 H=2666:00 +CPU= 10.324
  18:33:26 STEP 2667 H=2667:00 +CPU= 16.457
  18:33:37 STEP 2668 H=2668:00 +CPU= 10.977
  18:33:47 STEP 2669 H=2669:00 +CPU= 10.064
  18:34:04 STEP 2670 H=2670:00 +CPU= 16.854
  18:34:14 STEP 2671 H=2671:00 +CPU= 10.135
  18:34:24 STEP 2672 H=2672:00 +CPU= 10.273
  18:34:41 STEP 2673 H=2673:00 +CPU= 16.743
  18:34:51 STEP 2674 H=2674:00 +CPU= 10.413
  18:35:01 STEP 2675 H=2675:00 +CPU= 10.151
  18:35:19 STEP 2676 H=2676:00 +CPU= 17.953
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002676
  18:35:29 STEP 2677 H=2677:00 +CPU=  9.816
  18:35:40 STEP 2678 H=2678:00 +CPU= 10.777
  18:35:56 STEP 2679 H=2679:00 +CPU= 16.150
  18:36:07 STEP 2680 H=2680:00 +CPU= 10.736
  18:36:17 STEP 2681 H=2681:00 +CPU= 10.129
  18:36:33 STEP 2682 H=2682:00 +CPU= 16.428
  18:36:44 STEP 2683 H=2683:00 +CPU= 10.364
  18:36:54 STEP 2684 H=2684:00 +CPU= 10.265
  18:37:10 STEP 2685 H=2685:00 +CPU= 16.268
  18:37:21 STEP 2686 H=2686:00 +CPU= 10.538
  18:37:31 STEP 2687 H=2687:00 +CPU= 10.406
  18:37:50 STEP 2688 H=2688:00 +CPU= 19.122
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_111.zip
Uploading the intermediate file: upload_file_111.zip
  18:38:01 STEP 2689 H=2689:00 +CPU= 10.117
  18:38:11 STEP 2690 H=2690:00 +CPU= 10.480
Uploading trickle at timestep: 9676800
  18:38:27 STEP 2691 H=2691:00 +CPU= 16.264
  18:38:38 STEP 2692 H=2692:00 +CPU= 10.496
  18:38:48 STEP 2693 H=2693:00 +CPU=  9.943
  18:39:04 STEP 2694 H=2694:00 +CPU= 16.542
  18:39:15 STEP 2695 H=2695:00 +CPU= 10.474
  18:39:25 STEP 2696 H=2696:00 +CPU= 10.332
  18:39:42 STEP 2697 H=2697:00 +CPU= 16.720
  18:39:52 STEP 2698 H=2698:00 +CPU= 10.565
  18:40:02 STEP 2699 H=2699:00 +CPU= 10.008
  18:40:21 STEP 2700 H=2700:00 +CPU= 18.283
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002700
  18:40:31 STEP 2701 H=2701:00 +CPU=  9.907
  18:40:42 STEP 2702 H=2702:00 +CPU= 11.012
  18:40:58 STEP 2703 H=2703:00 +CPU= 16.534
  18:41:09 STEP 2704 H=2704:00 +CPU= 10.635
  18:41:19 STEP 2705 H=2705:00 +CPU=  9.874
  18:41:35 STEP 2706 H=2706:00 +CPU= 16.602
  18:41:46 STEP 2707 H=2707:00 +CPU= 10.308
  18:41:56 STEP 2708 H=2708:00 +CPU= 10.352
  18:42:12 STEP 2709 H=2709:00 +CPU= 16.588
  18:42:23 STEP 2710 H=2710:00 +CPU= 10.401
  18:42:33 STEP 2711 H=2711:00 +CPU= 10.449
  18:42:53 STEP 2712 H=2712:00 +CPU= 19.352
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_112.zip
Uploading the intermediate file: upload_file_112.zip
  18:43:03 STEP 2713 H=2713:00 +CPU= 10.044
  18:43:13 STEP 2714 H=2714:00 +CPU= 10.510
  18:43:30 STEP 2715 H=2715:00 +CPU= 16.663
  18:43:41 STEP 2716 H=2716:00 +CPU= 10.625
  18:43:51 STEP 2717 H=2717:00 +CPU= 10.300
  18:44:08 STEP 2718 H=2718:00 +CPU= 17.046
  18:44:18 STEP 2719 H=2719:00 +CPU= 10.642
  18:44:29 STEP 2720 H=2720:00 +CPU= 10.508
  18:44:46 STEP 2721 H=2721:00 +CPU= 16.736
  18:44:56 STEP 2722 H=2722:00 +CPU= 10.392
  18:45:06 STEP 2723 H=2723:00 +CPU=  9.897
  18:45:24 STEP 2724 H=2724:00 +CPU= 18.364
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002724
  18:45:34 STEP 2725 H=2725:00 +CPU= 10.055
  18:45:45 STEP 2726 H=2726:00 +CPU= 10.992
  18:46:02 STEP 2727 H=2727:00 +CPU= 16.380
  18:46:12 STEP 2728 H=2728:00 +CPU= 10.380
  18:46:23 STEP 2729 H=2729:00 +CPU= 10.414
  18:46:39 STEP 2730 H=2730:00 +CPU= 16.667
  18:46:50 STEP 2731 H=2731:00 +CPU= 10.665
  18:47:00 STEP 2732 H=2732:00 +CPU= 10.463
  18:47:17 STEP 2733 H=2733:00 +CPU= 16.370
  18:47:27 STEP 2734 H=2734:00 +CPU= 10.314
  18:47:37 STEP 2735 H=2735:00 +CPU= 10.094
  18:47:56 STEP 2736 H=2736:00 +CPU= 19.291
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_113.zip
Uploading the intermediate file: upload_file_113.zip
  18:48:07 STEP 2737 H=2737:00 +CPU= 10.049
  18:48:17 STEP 2738 H=2738:00 +CPU= 10.183
  18:48:33 STEP 2739 H=2739:00 +CPU= 16.472
  18:48:44 STEP 2740 H=2740:00 +CPU= 10.393
  18:48:54 STEP 2741 H=2741:00 +CPU= 10.303
  18:49:11 STEP 2742 H=2742:00 +CPU= 16.708
  18:49:21 STEP 2743 H=2743:00 +CPU= 10.401
  18:49:31 STEP 2744 H=2744:00 +CPU= 10.364
  18:49:48 STEP 2745 H=2745:00 +CPU= 16.559
  18:49:58 STEP 2746 H=2746:00 +CPU= 10.443
  18:50:09 STEP 2747 H=2747:00 +CPU= 10.254
  18:50:27 STEP 2748 H=2748:00 +CPU= 18.311
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002748
  18:50:37 STEP 2749 H=2749:00 +CPU= 10.204
  18:50:47 STEP 2750 H=2750:00 +CPU= 10.355
  18:51:04 STEP 2751 H=2751:00 +CPU= 16.679
  18:51:15 STEP 2752 H=2752:00 +CPU= 10.524
  18:51:25 STEP 2753 H=2753:00 +CPU= 10.176
  18:51:41 STEP 2754 H=2754:00 +CPU= 16.430
  18:51:51 STEP 2755 H=2755:00 +CPU= 10.170
  18:52:02 STEP 2756 H=2756:00 +CPU= 10.844
  18:52:19 STEP 2757 H=2757:00 +CPU= 16.415
  18:52:29 STEP 2758 H=2758:00 +CPU= 10.543
  18:52:40 STEP 2759 H=2759:00 +CPU= 10.272
  18:52:59 STEP 2760 H=2760:00 +CPU= 19.573
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_114.zip
Uploading the intermediate file: upload_file_114.zip
  18:53:10 STEP 2761 H=2761:00 +CPU= 10.331
  18:53:20 STEP 2762 H=2762:00 +CPU= 10.469
  18:53:36 STEP 2763 H=2763:00 +CPU= 16.254
  18:53:47 STEP 2764 H=2764:00 +CPU= 10.409
  18:53:57 STEP 2765 H=2765:00 +CPU= 10.027
  18:54:14 STEP 2766 H=2766:00 +CPU= 16.935
  18:54:24 STEP 2767 H=2767:00 +CPU= 10.262
  18:54:35 STEP 2768 H=2768:00 +CPU= 11.345
  18:54:52 STEP 2769 H=2769:00 +CPU= 16.240
  18:55:02 STEP 2770 H=2770:00 +CPU= 10.596
  18:55:12 STEP 2771 H=2771:00 +CPU=  9.946
  18:55:30 STEP 2772 H=2772:00 +CPU= 17.721
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002772
  18:55:41 STEP 2773 H=2773:00 +CPU= 10.783
  18:55:51 STEP 2774 H=2774:00 +CPU= 10.658
  18:56:08 STEP 2775 H=2775:00 +CPU= 16.456
  18:56:18 STEP 2776 H=2776:00 +CPU= 10.496
  18:56:28 STEP 2777 H=2777:00 +CPU= 10.087
  18:56:45 STEP 2778 H=2778:00 +CPU= 17.111
  18:56:56 STEP 2779 H=2779:00 +CPU= 10.520
  18:57:07 STEP 2780 H=2780:00 +CPU= 10.936
  18:57:24 STEP 2781 H=2781:00 +CPU= 16.835
  18:57:34 STEP 2782 H=2782:00 +CPU= 10.276
  18:57:45 STEP 2783 H=2783:00 +CPU= 10.509
  18:58:04 STEP 2784 H=2784:00 +CPU= 19.090
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_115.zip
Uploading the intermediate file: upload_file_115.zip
  18:58:14 STEP 2785 H=2785:00 +CPU= 10.234
  18:58:24 STEP 2786 H=2786:00 +CPU= 10.385
  18:58:41 STEP 2787 H=2787:00 +CPU= 16.289
  18:58:51 STEP 2788 H=2788:00 +CPU= 10.476
  18:59:01 STEP 2789 H=2789:00 +CPU=  9.981
  18:59:18 STEP 2790 H=2790:00 +CPU= 16.906
  18:59:28 STEP 2791 H=2791:00 +CPU= 10.256
  18:59:38 STEP 2792 H=2792:00 +CPU= 10.313
  18:59:55 STEP 2793 H=2793:00 +CPU= 16.556
  19:00:05 STEP 2794 H=2794:00 +CPU= 10.286
  19:00:16 STEP 2795 H=2795:00 +CPU= 10.862
  19:00:34 STEP 2796 H=2796:00 +CPU= 17.936
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002796
  19:00:44 STEP 2797 H=2797:00 +CPU= 10.216
  19:00:55 STEP 2798 H=2798:00 +CPU= 10.408
  19:01:11 STEP 2799 H=2799:00 +CPU= 16.301
  19:01:22 STEP 2800 H=2800:00 +CPU= 10.686
  19:01:32 STEP 2801 H=2801:00 +CPU= 10.085
  19:01:49 STEP 2802 H=2802:00 +CPU= 16.806
  19:01:59 STEP 2803 H=2803:00 +CPU= 10.160
  19:02:09 STEP 2804 H=2804:00 +CPU= 10.176
  19:02:26 STEP 2805 H=2805:00 +CPU= 16.719
  19:02:36 STEP 2806 H=2806:00 +CPU= 10.459
  19:02:47 STEP 2807 H=2807:00 +CPU= 10.743
  19:03:06 STEP 2808 H=2808:00 +CPU= 18.907
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_116.zip
Uploading the intermediate file: upload_file_116.zip
  19:03:16 STEP 2809 H=2809:00 +CPU= 10.188
  19:03:27 STEP 2810 H=2810:00 +CPU= 10.517
  19:03:43 STEP 2811 H=2811:00 +CPU= 16.327
  19:03:53 STEP 2812 H=2812:00 +CPU= 10.361
  19:04:04 STEP 2813 H=2813:00 +CPU= 10.468
  19:04:20 STEP 2814 H=2814:00 +CPU= 16.781
  19:04:31 STEP 2815 H=2815:00 +CPU= 10.317
  19:04:41 STEP 2816 H=2816:00 +CPU= 10.328
  19:04:58 STEP 2817 H=2817:00 +CPU= 16.997
  19:05:09 STEP 2818 H=2818:00 +CPU= 10.492
  19:05:19 STEP 2819 H=2819:00 +CPU= 10.374
  19:05:37 STEP 2820 H=2820:00 +CPU= 18.152
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002820
  19:05:47 STEP 2821 H=2821:00 +CPU= 10.076
  19:05:58 STEP 2822 H=2822:00 +CPU= 11.105
  19:06:14 STEP 2823 H=2823:00 +CPU= 16.169
  19:06:25 STEP 2824 H=2824:00 +CPU= 10.416
  19:06:35 STEP 2825 H=2825:00 +CPU= 10.044
  19:06:52 STEP 2826 H=2826:00 +CPU= 16.675
  19:07:02 STEP 2827 H=2827:00 +CPU= 10.125
  19:07:12 STEP 2828 H=2828:00 +CPU= 10.225
  19:07:29 STEP 2829 H=2829:00 +CPU= 16.583
  19:07:39 STEP 2830 H=2830:00 +CPU= 10.247
  19:07:49 STEP 2831 H=2831:00 +CPU= 10.307
  19:08:08 STEP 2832 H=2832:00 +CPU= 19.334
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_117.zip
Uploading the intermediate file: upload_file_117.zip
  19:08:19 STEP 2833 H=2833:00 +CPU= 10.085
  19:08:29 STEP 2834 H=2834:00 +CPU= 10.718
  19:08:46 STEP 2835 H=2835:00 +CPU= 16.341
  19:08:56 STEP 2836 H=2836:00 +CPU= 10.276
  19:09:07 STEP 2837 H=2837:00 +CPU= 10.792
  19:09:23 STEP 2838 H=2838:00 +CPU= 16.680
  19:09:34 STEP 2839 H=2839:00 +CPU= 10.199
  19:09:44 STEP 2840 H=2840:00 +CPU= 10.419
  19:10:00 STEP 2841 H=2841:00 +CPU= 16.389
  19:10:11 STEP 2842 H=2842:00 +CPU= 10.177
  19:10:21 STEP 2843 H=2843:00 +CPU= 10.786
  19:10:40 STEP 2844 H=2844:00 +CPU= 18.372
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002844
  19:10:50 STEP 2845 H=2845:00 +CPU= 10.054
  19:11:00 STEP 2846 H=2846:00 +CPU= 10.290
  19:11:17 STEP 2847 H=2847:00 +CPU= 16.544
  19:11:27 STEP 2848 H=2848:00 +CPU= 10.284
  19:11:38 STEP 2849 H=2849:00 +CPU= 11.116
  19:11:54 STEP 2850 H=2850:00 +CPU= 16.456
  19:12:05 STEP 2851 H=2851:00 +CPU= 10.193
  19:12:15 STEP 2852 H=2852:00 +CPU= 10.784
  19:12:32 STEP 2853 H=2853:00 +CPU= 16.256
  19:12:42 STEP 2854 H=2854:00 +CPU= 10.282
  19:12:52 STEP 2855 H=2855:00 +CPU= 10.383
  19:13:12 STEP 2856 H=2856:00 +CPU= 19.162
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_118.zip
Uploading the intermediate file: upload_file_118.zip
  19:13:22 STEP 2857 H=2857:00 +CPU= 10.174
  19:13:32 STEP 2858 H=2858:00 +CPU= 10.378
  19:13:49 STEP 2859 H=2859:00 +CPU= 16.856
  19:13:59 STEP 2860 H=2860:00 +CPU= 10.425
  19:14:10 STEP 2861 H=2861:00 +CPU= 10.236
  19:14:27 STEP 2862 H=2862:00 +CPU= 17.083
  19:14:37 STEP 2863 H=2863:00 +CPU= 10.787
  19:14:48 STEP 2864 H=2864:00 +CPU= 10.979
  19:15:05 STEP 2865 H=2865:00 +CPU= 16.955
  19:15:16 STEP 2866 H=2866:00 +CPU= 10.877
  19:15:27 STEP 2867 H=2867:00 +CPU= 10.781
  19:15:46 STEP 2868 H=2868:00 +CPU= 19.161
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002868
  19:15:57 STEP 2869 H=2869:00 +CPU= 10.804
  19:16:08 STEP 2870 H=2870:00 +CPU= 11.409
  19:16:26 STEP 2871 H=2871:00 +CPU= 17.225
  19:16:36 STEP 2872 H=2872:00 +CPU= 10.771
  19:16:48 STEP 2873 H=2873:00 +CPU= 11.058
  19:17:05 STEP 2874 H=2874:00 +CPU= 17.735
  19:17:16 STEP 2875 H=2875:00 +CPU= 10.684
  19:17:27 STEP 2876 H=2876:00 +CPU= 10.960
  19:17:44 STEP 2877 H=2877:00 +CPU= 17.327
  19:17:55 STEP 2878 H=2878:00 +CPU= 11.137
  19:18:06 STEP 2879 H=2879:00 +CPU= 11.071
  19:18:27 STEP 2880 H=2880:00 +CPU= 20.188
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_119.zip
Uploading the intermediate file: upload_file_119.zip
  19:18:37 STEP 2881 H=2881:00 +CPU= 10.098
  19:18:48 STEP 2882 H=2882:00 +CPU= 11.005
  19:19:04 STEP 2883 H=2883:00 +CPU= 16.286
  19:19:14 STEP 2884 H=2884:00 +CPU= 10.250
  19:19:25 STEP 2885 H=2885:00 +CPU= 10.470
  19:19:42 STEP 2886 H=2886:00 +CPU= 16.792
  19:19:52 STEP 2887 H=2887:00 +CPU= 10.031
  19:20:02 STEP 2888 H=2888:00 +CPU= 10.786
  19:20:19 STEP 2889 H=2889:00 +CPU= 16.707
  19:20:30 STEP 2890 H=2890:00 +CPU= 10.354
  19:20:40 STEP 2891 H=2891:00 +CPU= 10.136
  19:20:59 STEP 2892 H=2892:00 +CPU= 18.911
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002892
  19:21:09 STEP 2893 H=2893:00 +CPU= 10.169
  19:21:19 STEP 2894 H=2894:00 +CPU= 10.339
  19:21:36 STEP 2895 H=2895:00 +CPU= 16.742
  19:21:46 STEP 2896 H=2896:00 +CPU= 10.327
  19:21:57 STEP 2897 H=2897:00 +CPU= 10.441
  19:22:13 STEP 2898 H=2898:00 +CPU= 16.734
  19:22:23 STEP 2899 H=2899:00 +CPU= 10.040
  19:22:34 STEP 2900 H=2900:00 +CPU= 11.109
  19:22:51 STEP 2901 H=2901:00 +CPU= 16.253
  19:23:01 STEP 2902 H=2902:00 +CPU= 10.632
  19:23:12 STEP 2903 H=2903:00 +CPU= 10.671
  19:23:32 STEP 2904 H=2904:00 +CPU= 19.604
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_120.zip
Uploading the intermediate file: upload_file_120.zip
  19:23:42 STEP 2905 H=2905:00 +CPU= 10.254
  19:23:52 STEP 2906 H=2906:00 +CPU= 10.339
  19:24:09 STEP 2907 H=2907:00 +CPU= 16.902
  19:24:19 STEP 2908 H=2908:00 +CPU= 10.253
  19:24:30 STEP 2909 H=2909:00 +CPU= 10.310
  19:24:47 STEP 2910 H=2910:00 +CPU= 17.148
  19:24:57 STEP 2911 H=2911:00 +CPU= 10.152
  19:25:07 STEP 2912 H=2912:00 +CPU= 10.386
  19:25:24 STEP 2913 H=2913:00 +CPU= 16.636
  19:25:34 STEP 2914 H=2914:00 +CPU= 10.376
  19:25:44 STEP 2915 H=2915:00 +CPU= 10.003
  19:26:03 STEP 2916 H=2916:00 +CPU= 18.577
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002916
  19:26:13 STEP 2917 H=2917:00 +CPU= 10.222
  19:26:24 STEP 2918 H=2918:00 +CPU= 10.591
  19:26:40 STEP 2919 H=2919:00 +CPU= 16.478
  19:26:51 STEP 2920 H=2920:00 +CPU= 10.404
  19:27:01 STEP 2921 H=2921:00 +CPU= 10.733
  19:27:18 STEP 2922 H=2922:00 +CPU= 16.710
  19:27:28 STEP 2923 H=2923:00 +CPU= 10.311
  19:27:39 STEP 2924 H=2924:00 +CPU= 10.801
  19:27:55 STEP 2925 H=2925:00 +CPU= 16.111
  19:28:06 STEP 2926 H=2926:00 +CPU= 10.605
  19:28:16 STEP 2927 H=2927:00 +CPU= 10.242
  19:28:35 STEP 2928 H=2928:00 +CPU= 18.191
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_121.zip
Uploading the intermediate file: upload_file_121.zip
  19:28:44 STEP 2929 H=2929:00 +CPU=  9.530
  19:28:54 STEP 2930 H=2930:00 +CPU=  9.623
Uploading trickle at timestep: 10540800
  19:29:09 STEP 2931 H=2931:00 +CPU= 15.124
  19:29:18 STEP 2932 H=2932:00 +CPU=  9.466
  19:29:28 STEP 2933 H=2933:00 +CPU=  9.730
  19:29:43 STEP 2934 H=2934:00 +CPU= 15.373
  19:29:53 STEP 2935 H=2935:00 +CPU=  9.391
  19:30:02 STEP 2936 H=2936:00 +CPU=  9.707
  19:30:19 STEP 2937 H=2937:00 +CPU= 16.129
  19:30:28 STEP 2938 H=2938:00 +CPU=  9.828
  19:30:38 STEP 2939 H=2939:00 +CPU=  9.210
  19:30:54 STEP 2940 H=2940:00 +CPU= 16.833
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002940
  19:31:04 STEP 2941 H=2941:00 +CPU=  9.434
  19:31:13 STEP 2942 H=2942:00 +CPU=  9.579
  19:31:30 STEP 2943 H=2943:00 +CPU= 16.106
  19:31:39 STEP 2944 H=2944:00 +CPU=  9.492
  19:31:48 STEP 2945 H=2945:00 +CPU=  9.354
  19:32:05 STEP 2946 H=2946:00 +CPU= 16.794
  19:32:15 STEP 2947 H=2947:00 +CPU=  9.986
  19:32:26 STEP 2948 H=2948:00 +CPU= 10.272
  19:32:42 STEP 2949 H=2949:00 +CPU= 16.339
  19:32:52 STEP 2950 H=2950:00 +CPU= 10.423
  19:33:02 STEP 2951 H=2951:00 +CPU= 10.220
  19:33:21 STEP 2952 H=2952:00 +CPU= 18.635
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12186200/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0556_2000050100_123_969_12186200_0_r583443451_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
19:36:44 (338): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:30:06 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 7,084,800 37,001 0.0052
09 May 2023 16:30:01 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 6,048,000 33,168 0.0055
09 May 2023 16:29:53 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 5,184,000 29,185 0.0056
09 May 2023 16:29:47 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 4,320,000 23,389 0.0054
09 May 2023 16:29:41 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 3,456,000 19,074 0.0055
09 May 2023 16:29:30 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 2,592,000 13,360 0.0052
09 May 2023 16:29:24 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 1,728,000 8,538 0.0049
09 May 2023 16:29:13 1537625 22271563 oifs_43r3_ps_0556_2000050100_123_969_12186200_0 864,000 3,490 0.0040


©2024 climateprediction.net