climateprediction.net home page
Task 22300072

Task 22300072

Name oifs_43r3_ps_0130_2011050100_123_980_12196774_1
Workunit 12196774
Created 19 Jan 2023, 22:30:40 UTC
Sent 19 Jan 2023, 22:53:34 UTC
Report deadline 20 Mar 2023, 22:53:34 UTC
Received 21 Jan 2023, 1:59:48 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537625
Run time 16 hours 58 min 20 sec
CPU time 16 hours 54 min 28 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,799.71 MB
Peak swap size 4,977.80 MB
Peak disk usage 2,044.65 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
 the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_94.zip
Uploading the intermediate file: upload_file_94.zip
  09:19:45 STEP 2281 H=2281:00 +CPU= 17.672
  09:20:02 STEP 2282 H=2282:00 +CPU= 17.802
  09:20:32 STEP 2283 H=2283:00 +CPU= 30.027
  09:20:51 STEP 2284 H=2284:00 +CPU= 18.108
  09:21:08 STEP 2285 H=2285:00 +CPU= 17.359
  09:21:37 STEP 2286 H=2286:00 +CPU= 28.615
  09:21:54 STEP 2287 H=2287:00 +CPU= 17.927
  09:22:13 STEP 2288 H=2288:00 +CPU= 18.151
  09:22:41 STEP 2289 H=2289:00 +CPU= 28.432
  09:23:00 STEP 2290 H=2290:00 +CPU= 18.581
  09:23:18 STEP 2291 H=2291:00 +CPU= 17.865
  09:23:48 STEP 2292 H=2292:00 +CPU= 30.724
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002292
  09:24:06 STEP 2293 H=2293:00 +CPU= 17.553
  09:24:24 STEP 2294 H=2294:00 +CPU= 18.080
  09:24:52 STEP 2295 H=2295:00 +CPU= 27.846
  09:25:10 STEP 2296 H=2296:00 +CPU= 18.401
  09:25:30 STEP 2297 H=2297:00 +CPU= 19.364
  09:25:58 STEP 2298 H=2298:00 +CPU= 28.524
  09:26:16 STEP 2299 H=2299:00 +CPU= 17.759
  09:26:34 STEP 2300 H=2300:00 +CPU= 18.171
  09:27:01 STEP 2301 H=2301:00 +CPU= 27.373
  09:27:19 STEP 2302 H=2302:00 +CPU= 18.033
  09:27:37 STEP 2303 H=2303:00 +CPU= 18.020
  09:28:13 STEP 2304 H=2304:00 +CPU= 35.386
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_95.zip
Uploading the intermediate file: upload_file_95.zip
  09:28:30 STEP 2305 H=2305:00 +CPU= 17.674
Uploading trickle at timestep: 8294400
  09:28:49 STEP 2306 H=2306:00 +CPU= 18.323
  09:29:16 STEP 2307 H=2307:00 +CPU= 27.391
  09:29:34 STEP 2308 H=2308:00 +CPU= 17.945
  09:29:52 STEP 2309 H=2309:00 +CPU= 17.451
  09:30:22 STEP 2310 H=2310:00 +CPU= 29.981
  09:30:39 STEP 2311 H=2311:00 +CPU= 17.749
  09:30:57 STEP 2312 H=2312:00 +CPU= 17.600
  09:31:25 STEP 2313 H=2313:00 +CPU= 28.197
  09:31:43 STEP 2314 H=2314:00 +CPU= 17.978
  09:32:02 STEP 2315 H=2315:00 +CPU= 18.399
  09:32:32 STEP 2316 H=2316:00 +CPU= 30.531
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002316
  09:32:51 STEP 2317 H=2317:00 +CPU= 18.487
  09:33:08 STEP 2318 H=2318:00 +CPU= 17.875
  09:33:36 STEP 2319 H=2319:00 +CPU= 27.432
  09:33:54 STEP 2320 H=2320:00 +CPU= 18.572
  09:34:12 STEP 2321 H=2321:00 +CPU= 17.282
  09:34:40 STEP 2322 H=2322:00 +CPU= 27.887
  09:34:57 STEP 2323 H=2323:00 +CPU= 17.635
  09:35:15 STEP 2324 H=2324:00 +CPU= 18.201
  09:35:43 STEP 2325 H=2325:00 +CPU= 27.592
  09:36:01 STEP 2326 H=2326:00 +CPU= 18.273
  09:36:20 STEP 2327 H=2327:00 +CPU= 18.161
  09:36:54 STEP 2328 H=2328:00 +CPU= 34.555
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_96.zip
Uploading the intermediate file: upload_file_96.zip
  09:37:12 STEP 2329 H=2329:00 +CPU= 17.812
  09:37:30 STEP 2330 H=2330:00 +CPU= 18.127
  09:37:58 STEP 2331 H=2331:00 +CPU= 28.266
  09:38:17 STEP 2332 H=2332:00 +CPU= 19.012
  09:38:37 STEP 2333 H=2333:00 +CPU= 19.671
  09:39:06 STEP 2334 H=2334:00 +CPU= 29.210
  09:39:25 STEP 2335 H=2335:00 +CPU= 18.375
  09:39:43 STEP 2336 H=2336:00 +CPU= 18.892
  09:40:12 STEP 2337 H=2337:00 +CPU= 28.513
  09:40:30 STEP 2338 H=2338:00 +CPU= 18.340
  09:40:49 STEP 2339 H=2339:00 +CPU= 18.199
  09:41:20 STEP 2340 H=2340:00 +CPU= 31.954
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002340
  09:41:39 STEP 2341 H=2341:00 +CPU= 18.265
  09:41:58 STEP 2342 H=2342:00 +CPU= 19.228
  09:42:27 STEP 2343 H=2343:00 +CPU= 28.571
  09:42:45 STEP 2344 H=2344:00 +CPU= 18.897
  09:43:04 STEP 2345 H=2345:00 +CPU= 18.529
  09:43:33 STEP 2346 H=2346:00 +CPU= 29.000
  09:43:51 STEP 2347 H=2347:00 +CPU= 17.926
  09:44:09 STEP 2348 H=2348:00 +CPU= 18.253
  09:44:37 STEP 2349 H=2349:00 +CPU= 27.617
  09:44:55 STEP 2350 H=2350:00 +CPU= 18.011
  09:45:13 STEP 2351 H=2351:00 +CPU= 18.165
  09:45:46 STEP 2352 H=2352:00 +CPU= 32.709
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_97.zip
Uploading the intermediate file: upload_file_97.zip
  09:46:04 STEP 2353 H=2353:00 +CPU= 18.070
  09:46:24 STEP 2354 H=2354:00 +CPU= 20.058
  09:46:52 STEP 2355 H=2355:00 +CPU= 28.418
  09:47:11 STEP 2356 H=2356:00 +CPU= 18.382
  09:47:28 STEP 2357 H=2357:00 +CPU= 17.565
  09:47:57 STEP 2358 H=2358:00 +CPU= 28.379
  09:48:15 STEP 2359 H=2359:00 +CPU= 18.007
  09:48:33 STEP 2360 H=2360:00 +CPU= 17.992
  09:49:00 STEP 2361 H=2361:00 +CPU= 27.534
  09:49:19 STEP 2362 H=2362:00 +CPU= 18.353
  09:49:36 STEP 2363 H=2363:00 +CPU= 17.840
  09:50:08 STEP 2364 H=2364:00 +CPU= 31.135
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002364
  09:50:27 STEP 2365 H=2365:00 +CPU= 19.915
  09:50:47 STEP 2366 H=2366:00 +CPU= 19.293
  09:51:15 STEP 2367 H=2367:00 +CPU= 28.302
  09:51:34 STEP 2368 H=2368:00 +CPU= 18.970
  09:51:52 STEP 2369 H=2369:00 +CPU= 17.961
  09:52:20 STEP 2370 H=2370:00 +CPU= 28.467
  09:52:38 STEP 2371 H=2371:00 +CPU= 17.818
  09:52:57 STEP 2372 H=2372:00 +CPU= 18.309
  09:53:25 STEP 2373 H=2373:00 +CPU= 28.093
  09:53:43 STEP 2374 H=2374:00 +CPU= 18.092
  09:54:01 STEP 2375 H=2375:00 +CPU= 18.219
  09:54:36 STEP 2376 H=2376:00 +CPU= 34.569
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_98.zip
Uploading the intermediate file: upload_file_98.zip
  09:54:53 STEP 2377 H=2377:00 +CPU= 17.859
  09:55:11 STEP 2378 H=2378:00 +CPU= 18.094
  09:55:40 STEP 2379 H=2379:00 +CPU= 28.467
  09:55:58 STEP 2380 H=2380:00 +CPU= 18.283
  09:56:16 STEP 2381 H=2381:00 +CPU= 17.777
  09:56:44 STEP 2382 H=2382:00 +CPU= 28.032
  09:57:02 STEP 2383 H=2383:00 +CPU= 17.753
  09:57:20 STEP 2384 H=2384:00 +CPU= 18.161
  09:57:48 STEP 2385 H=2385:00 +CPU= 27.741
  09:58:06 STEP 2386 H=2386:00 +CPU= 18.100
  09:58:24 STEP 2387 H=2387:00 +CPU= 17.908
  09:58:55 STEP 2388 H=2388:00 +CPU= 31.244
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002388
  09:59:14 STEP 2389 H=2389:00 +CPU= 18.519
  09:59:33 STEP 2390 H=2390:00 +CPU= 19.062
  10:00:01 STEP 2391 H=2391:00 +CPU= 28.148
  10:00:19 STEP 2392 H=2392:00 +CPU= 18.611
  10:00:37 STEP 2393 H=2393:00 +CPU= 18.115
  10:01:06 STEP 2394 H=2394:00 +CPU= 28.334
  10:01:24 STEP 2395 H=2395:00 +CPU= 17.995
  10:01:44 STEP 2396 H=2396:00 +CPU= 19.833
  10:02:12 STEP 2397 H=2397:00 +CPU= 28.461
  10:02:31 STEP 2398 H=2398:00 +CPU= 18.664
  10:02:49 STEP 2399 H=2399:00 +CPU= 18.002
  10:03:22 STEP 2400 H=2400:00 +CPU= 32.771
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_99.zip
Uploading the intermediate file: upload_file_99.zip
  10:03:40 STEP 2401 H=2401:00 +CPU= 17.964
  10:03:58 STEP 2402 H=2402:00 +CPU= 18.630
  10:04:26 STEP 2403 H=2403:00 +CPU= 27.632
  10:04:45 STEP 2404 H=2404:00 +CPU= 19.371
  10:05:03 STEP 2405 H=2405:00 +CPU= 17.667
  10:05:32 STEP 2406 H=2406:00 +CPU= 28.825
  10:05:49 STEP 2407 H=2407:00 +CPU= 17.687
  10:06:07 STEP 2408 H=2408:00 +CPU= 17.324
  10:06:35 STEP 2409 H=2409:00 +CPU= 28.266
  10:06:53 STEP 2410 H=2410:00 +CPU= 18.282
  10:07:10 STEP 2411 H=2411:00 +CPU= 16.994
  10:07:40 STEP 2412 H=2412:00 +CPU= 29.576
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002412
  10:07:58 STEP 2413 H=2413:00 +CPU= 17.693
  10:08:16 STEP 2414 H=2414:00 +CPU= 18.057
  10:08:43 STEP 2415 H=2415:00 +CPU= 27.514
  10:09:02 STEP 2416 H=2416:00 +CPU= 18.370
  10:09:19 STEP 2417 H=2417:00 +CPU= 17.470
  10:09:48 STEP 2418 H=2418:00 +CPU= 28.939
  10:10:07 STEP 2419 H=2419:00 +CPU= 19.355
  10:10:25 STEP 2420 H=2420:00 +CPU= 17.728
  10:10:52 STEP 2421 H=2421:00 +CPU= 27.320
  10:11:10 STEP 2422 H=2422:00 +CPU= 17.907
  10:11:28 STEP 2423 H=2423:00 +CPU= 17.877
  10:12:01 STEP 2424 H=2424:00 +CPU= 32.386
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_100.zip
Uploading the intermediate file: upload_file_100.zip
  10:12:18 STEP 2425 H=2425:00 +CPU= 17.508
  10:12:37 STEP 2426 H=2426:00 +CPU= 18.997
  10:13:06 STEP 2427 H=2427:00 +CPU= 28.534
  10:13:24 STEP 2428 H=2428:00 +CPU= 18.680
  10:13:42 STEP 2429 H=2429:00 +CPU= 17.647
  10:14:10 STEP 2430 H=2430:00 +CPU= 28.322
  10:14:28 STEP 2431 H=2431:00 +CPU= 17.701
  10:14:46 STEP 2432 H=2432:00 +CPU= 18.035
  10:15:14 STEP 2433 H=2433:00 +CPU= 28.082
  10:15:32 STEP 2434 H=2434:00 +CPU= 17.829
  10:15:49 STEP 2435 H=2435:00 +CPU= 17.049
  10:16:19 STEP 2436 H=2436:00 +CPU= 29.826
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002436
  10:16:36 STEP 2437 H=2437:00 +CPU= 17.497
  10:16:54 STEP 2438 H=2438:00 +CPU= 17.403
  10:17:21 STEP 2439 H=2439:00 +CPU= 27.607
  10:17:40 STEP 2440 H=2440:00 +CPU= 18.170
  10:17:56 STEP 2441 H=2441:00 +CPU= 16.911
  10:18:23 STEP 2442 H=2442:00 +CPU= 26.788
  10:18:40 STEP 2443 H=2443:00 +CPU= 16.367
  10:18:57 STEP 2444 H=2444:00 +CPU= 17.001
  10:19:23 STEP 2445 H=2445:00 +CPU= 26.011
  10:19:39 STEP 2446 H=2446:00 +CPU= 16.791
  10:19:57 STEP 2447 H=2447:00 +CPU= 17.354
  10:20:28 STEP 2448 H=2448:00 +CPU= 30.821
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_101.zip
Uploading the intermediate file: upload_file_101.zip
  10:20:44 STEP 2449 H=2449:00 +CPU= 16.610
  10:21:03 STEP 2450 H=2450:00 +CPU= 18.801
  10:21:30 STEP 2451 H=2451:00 +CPU= 26.736
  10:21:47 STEP 2452 H=2452:00 +CPU= 17.643
  10:22:05 STEP 2453 H=2453:00 +CPU= 17.110
  10:22:32 STEP 2454 H=2454:00 +CPU= 27.225
  10:22:49 STEP 2455 H=2455:00 +CPU= 16.742
  10:23:06 STEP 2456 H=2456:00 +CPU= 17.683
  10:23:33 STEP 2457 H=2457:00 +CPU= 27.053
  10:23:51 STEP 2458 H=2458:00 +CPU= 17.855
  10:24:10 STEP 2459 H=2459:00 +CPU= 18.494
  10:24:39 STEP 2460 H=2460:00 +CPU= 29.820
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002460
  10:24:57 STEP 2461 H=2461:00 +CPU= 17.547
  10:25:15 STEP 2462 H=2462:00 +CPU= 17.743
  10:25:41 STEP 2463 H=2463:00 +CPU= 26.683
  10:25:58 STEP 2464 H=2464:00 +CPU= 16.830
  10:26:07 STEP 2465 H=2465:00 +CPU=  8.850
  10:26:22 STEP 2466 H=2466:00 +CPU= 14.638
  10:26:31 STEP 2467 H=2467:00 +CPU=  8.793
  10:26:40 STEP 2468 H=2468:00 +CPU=  9.001
  10:26:55 STEP 2469 H=2469:00 +CPU= 15.527
  10:27:04 STEP 2470 H=2470:00 +CPU=  8.789
  10:27:13 STEP 2471 H=2471:00 +CPU=  8.820
  10:27:29 STEP 2472 H=2472:00 +CPU= 16.291
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_102.zip
Uploading the intermediate file: upload_file_102.zip
  10:27:38 STEP 2473 H=2473:00 +CPU=  8.563
  10:27:46 STEP 2474 H=2474:00 +CPU=  8.684
  10:28:00 STEP 2475 H=2475:00 +CPU= 13.852
  10:28:09 STEP 2476 H=2476:00 +CPU=  8.716
  10:28:17 STEP 2477 H=2477:00 +CPU=  8.474
  10:28:31 STEP 2478 H=2478:00 +CPU= 14.224
  10:28:41 STEP 2479 H=2479:00 +CPU=  9.042
  10:28:59 STEP 2480 H=2480:00 +CPU= 18.578
  10:29:26 STEP 2481 H=2481:00 +CPU= 27.209
  10:29:44 STEP 2482 H=2482:00 +CPU= 17.710
  10:30:01 STEP 2483 H=2483:00 +CPU= 16.546
  10:30:30 STEP 2484 H=2484:00 +CPU= 29.035
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002484
  10:30:46 STEP 2485 H=2485:00 +CPU= 16.282
  10:31:03 STEP 2486 H=2486:00 +CPU= 16.725
  10:31:29 STEP 2487 H=2487:00 +CPU= 25.919
  10:31:45 STEP 2488 H=2488:00 +CPU= 16.499
  10:32:02 STEP 2489 H=2489:00 +CPU= 16.919
  10:32:28 STEP 2490 H=2490:00 +CPU= 26.495
  10:32:45 STEP 2491 H=2491:00 +CPU= 16.416
  10:33:02 STEP 2492 H=2492:00 +CPU= 16.745
  10:33:27 STEP 2493 H=2493:00 +CPU= 25.838
  10:33:45 STEP 2494 H=2494:00 +CPU= 17.160
  10:34:02 STEP 2495 H=2495:00 +CPU= 17.468
  10:34:32 STEP 2496 H=2496:00 +CPU= 30.364
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_103.zip
Uploading the intermediate file: upload_file_103.zip
  10:34:49 STEP 2497 H=2497:00 +CPU= 16.294
  10:35:06 STEP 2498 H=2498:00 +CPU= 16.883
  10:35:32 STEP 2499 H=2499:00 +CPU= 25.960
  10:35:48 STEP 2500 H=2500:00 +CPU= 16.723
  10:36:05 STEP 2501 H=2501:00 +CPU= 16.387
  10:36:33 STEP 2502 H=2502:00 +CPU= 27.786
  10:36:50 STEP 2503 H=2503:00 +CPU= 17.177
  10:37:07 STEP 2504 H=2504:00 +CPU= 17.148
  10:37:33 STEP 2505 H=2505:00 +CPU= 26.257
  10:37:50 STEP 2506 H=2506:00 +CPU= 16.729
  10:38:06 STEP 2507 H=2507:00 +CPU= 16.620
  10:38:35 STEP 2508 H=2508:00 +CPU= 28.549
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002508
  10:38:51 STEP 2509 H=2509:00 +CPU= 16.407
  10:39:08 STEP 2510 H=2510:00 +CPU= 17.035
  10:39:34 STEP 2511 H=2511:00 +CPU= 25.714
  10:39:51 STEP 2512 H=2512:00 +CPU= 17.169
  10:40:08 STEP 2513 H=2513:00 +CPU= 16.832
  10:40:35 STEP 2514 H=2514:00 +CPU= 26.525
  10:40:51 STEP 2515 H=2515:00 +CPU= 16.632
  10:41:09 STEP 2516 H=2516:00 +CPU= 17.146
  10:41:37 STEP 2517 H=2517:00 +CPU= 28.126
  10:41:54 STEP 2518 H=2518:00 +CPU= 17.043
  10:42:10 STEP 2519 H=2519:00 +CPU= 16.658
  10:42:41 STEP 2520 H=2520:00 +CPU= 30.160
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_104.zip
Uploading the intermediate file: upload_file_104.zip
  10:42:57 STEP 2521 H=2521:00 +CPU= 16.408
  10:43:14 STEP 2522 H=2522:00 +CPU= 16.628
  10:43:39 STEP 2523 H=2523:00 +CPU= 25.737
  10:43:56 STEP 2524 H=2524:00 +CPU= 16.505
  10:44:12 STEP 2525 H=2525:00 +CPU= 16.011
  10:44:38 STEP 2526 H=2526:00 +CPU= 25.940
  10:44:54 STEP 2527 H=2527:00 +CPU= 16.300
  10:45:11 STEP 2528 H=2528:00 +CPU= 16.478
  10:45:36 STEP 2529 H=2529:00 +CPU= 25.157
  10:45:52 STEP 2530 H=2530:00 +CPU= 16.476
  10:46:08 STEP 2531 H=2531:00 +CPU= 16.307
  10:46:34 STEP 2532 H=2532:00 +CPU= 25.676
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002532
  10:46:43 STEP 2533 H=2533:00 +CPU=  9.308
  10:46:53 STEP 2534 H=2534:00 +CPU=  9.385
  10:47:07 STEP 2535 H=2535:00 +CPU= 14.159
  10:47:16 STEP 2536 H=2536:00 +CPU=  9.146
  10:47:25 STEP 2537 H=2537:00 +CPU=  8.580
  10:47:39 STEP 2538 H=2538:00 +CPU= 14.474
  10:47:48 STEP 2539 H=2539:00 +CPU=  8.706
  10:47:57 STEP 2540 H=2540:00 +CPU=  8.870
  10:48:11 STEP 2541 H=2541:00 +CPU= 14.026
  10:48:20 STEP 2542 H=2542:00 +CPU=  8.826
  10:48:28 STEP 2543 H=2543:00 +CPU=  8.708
  10:48:45 STEP 2544 H=2544:00 +CPU= 16.099
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_105.zip
Uploading the intermediate file: upload_file_105.zip
  10:48:53 STEP 2545 H=2545:00 +CPU=  8.455
  10:49:02 STEP 2546 H=2546:00 +CPU=  8.723
Uploading trickle at timestep: 9158400
  10:49:16 STEP 2547 H=2547:00 +CPU= 13.710
  10:49:24 STEP 2548 H=2548:00 +CPU=  8.641
  10:49:33 STEP 2549 H=2549:00 +CPU=  8.434
  10:49:46 STEP 2550 H=2550:00 +CPU= 13.836
  10:49:55 STEP 2551 H=2551:00 +CPU=  8.421
  10:50:03 STEP 2552 H=2552:00 +CPU=  8.650
  10:50:18 STEP 2553 H=2553:00 +CPU= 14.203
  10:50:31 STEP 2554 H=2554:00 +CPU= 13.778
  10:50:47 STEP 2555 H=2555:00 +CPU= 15.822
  10:51:15 STEP 2556 H=2556:00 +CPU= 27.982
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002556
  10:51:31 STEP 2557 H=2557:00 +CPU= 15.794
  10:51:48 STEP 2558 H=2558:00 +CPU= 16.654
  10:52:13 STEP 2559 H=2559:00 +CPU= 25.592
  10:52:30 STEP 2560 H=2560:00 +CPU= 16.670
  10:52:46 STEP 2561 H=2561:00 +CPU= 15.828
  10:53:12 STEP 2562 H=2562:00 +CPU= 25.936
  10:53:28 STEP 2563 H=2563:00 +CPU= 15.987
  10:53:44 STEP 2564 H=2564:00 +CPU= 16.580
  10:54:12 STEP 2565 H=2565:00 +CPU= 27.469
  10:54:29 STEP 2566 H=2566:00 +CPU= 16.820
  10:54:46 STEP 2567 H=2567:00 +CPU= 16.994
  10:55:17 STEP 2568 H=2568:00 +CPU= 31.011
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_106.zip
Uploading the intermediate file: upload_file_106.zip
  10:55:33 STEP 2569 H=2569:00 +CPU= 16.319
  10:55:50 STEP 2570 H=2570:00 +CPU= 16.696
  10:56:16 STEP 2571 H=2571:00 +CPU= 25.954
  10:56:33 STEP 2572 H=2572:00 +CPU= 16.991
  10:56:50 STEP 2573 H=2573:00 +CPU= 17.228
  10:57:16 STEP 2574 H=2574:00 +CPU= 26.070
  10:57:33 STEP 2575 H=2575:00 +CPU= 16.813
  10:57:50 STEP 2576 H=2576:00 +CPU= 16.868
  10:58:15 STEP 2577 H=2577:00 +CPU= 25.813
  10:58:32 STEP 2578 H=2578:00 +CPU= 17.002
  10:58:49 STEP 2579 H=2579:00 +CPU= 16.094
  10:59:17 STEP 2580 H=2580:00 +CPU= 28.488
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002580
  10:59:33 STEP 2581 H=2581:00 +CPU= 15.523
  10:59:48 STEP 2582 H=2582:00 +CPU= 15.957
  11:00:13 STEP 2583 H=2583:00 +CPU= 24.847
  11:00:31 STEP 2584 H=2584:00 +CPU= 17.355
  11:00:46 STEP 2585 H=2585:00 +CPU= 15.478
  11:01:11 STEP 2586 H=2586:00 +CPU= 24.987
  11:01:27 STEP 2587 H=2587:00 +CPU= 15.761
  11:01:43 STEP 2588 H=2588:00 +CPU= 15.961
  11:02:08 STEP 2589 H=2589:00 +CPU= 24.835
  11:02:24 STEP 2590 H=2590:00 +CPU= 15.876
  11:02:40 STEP 2591 H=2591:00 +CPU= 16.043
  11:03:09 STEP 2592 H=2592:00 +CPU= 29.043
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_107.zip
Uploading the intermediate file: upload_file_107.zip
  11:03:25 STEP 2593 H=2593:00 +CPU= 16.016
  11:03:42 STEP 2594 H=2594:00 +CPU= 17.022
  11:04:06 STEP 2595 H=2595:00 +CPU= 24.707
  11:04:23 STEP 2596 H=2596:00 +CPU= 16.387
  11:04:38 STEP 2597 H=2597:00 +CPU= 15.434
  11:05:03 STEP 2598 H=2598:00 +CPU= 24.855
  11:05:19 STEP 2599 H=2599:00 +CPU= 15.742
  11:05:35 STEP 2600 H=2600:00 +CPU= 15.804
  11:05:59 STEP 2601 H=2601:00 +CPU= 24.654
  11:06:16 STEP 2602 H=2602:00 +CPU= 16.374
  11:06:32 STEP 2603 H=2603:00 +CPU= 15.789
  11:06:59 STEP 2604 H=2604:00 +CPU= 27.936
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002604
  11:07:16 STEP 2605 H=2605:00 +CPU= 16.071
  11:07:32 STEP 2606 H=2606:00 +CPU= 16.833
  11:07:58 STEP 2607 H=2607:00 +CPU= 25.383
  11:08:15 STEP 2608 H=2608:00 +CPU= 16.997
  11:08:31 STEP 2609 H=2609:00 +CPU= 16.727
  11:08:59 STEP 2610 H=2610:00 +CPU= 27.796
  11:09:16 STEP 2611 H=2611:00 +CPU= 16.284
  11:09:32 STEP 2612 H=2612:00 +CPU= 16.700
  11:09:58 STEP 2613 H=2613:00 +CPU= 25.949
  11:10:16 STEP 2614 H=2614:00 +CPU= 17.343
  11:10:33 STEP 2615 H=2615:00 +CPU= 17.031
  11:11:03 STEP 2616 H=2616:00 +CPU= 30.343
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_108.zip
Uploading the intermediate file: upload_file_108.zip
  11:11:19 STEP 2617 H=2617:00 +CPU= 16.296
  11:11:37 STEP 2618 H=2618:00 +CPU= 17.315
  11:12:04 STEP 2619 H=2619:00 +CPU= 27.566
  11:12:21 STEP 2620 H=2620:00 +CPU= 17.104
  11:12:38 STEP 2621 H=2621:00 +CPU= 16.268
  11:13:03 STEP 2622 H=2622:00 +CPU= 25.655
  11:13:19 STEP 2623 H=2623:00 +CPU= 16.022
  11:13:36 STEP 2624 H=2624:00 +CPU= 17.106
  11:14:02 STEP 2625 H=2625:00 +CPU= 25.650
  11:14:19 STEP 2626 H=2626:00 +CPU= 17.291
  11:14:36 STEP 2627 H=2627:00 +CPU= 16.640
  11:15:05 STEP 2628 H=2628:00 +CPU= 28.977
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002628
  11:15:22 STEP 2629 H=2629:00 +CPU= 16.990
  11:15:39 STEP 2630 H=2630:00 +CPU= 17.199
  11:16:05 STEP 2631 H=2631:00 +CPU= 26.098
  11:16:22 STEP 2632 H=2632:00 +CPU= 16.848
  11:16:39 STEP 2633 H=2633:00 +CPU= 16.388
  11:17:05 STEP 2634 H=2634:00 +CPU= 26.073
  11:17:22 STEP 2635 H=2635:00 +CPU= 16.407
  11:17:38 STEP 2636 H=2636:00 +CPU= 16.854
  11:18:05 STEP 2637 H=2637:00 +CPU= 26.074
  11:18:22 STEP 2638 H=2638:00 +CPU= 17.311
  11:18:39 STEP 2639 H=2639:00 +CPU= 16.797
  11:19:10 STEP 2640 H=2640:00 +CPU= 31.177
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_109.zip
Uploading the intermediate file: upload_file_109.zip
  11:19:26 STEP 2641 H=2641:00 +CPU= 16.236
  11:19:43 STEP 2642 H=2642:00 +CPU= 16.740
  11:20:09 STEP 2643 H=2643:00 +CPU= 26.301
  11:20:26 STEP 2644 H=2644:00 +CPU= 17.347
  11:20:45 STEP 2645 H=2645:00 +CPU= 18.407
  11:21:12 STEP 2646 H=2646:00 +CPU= 26.890
  11:21:29 STEP 2647 H=2647:00 +CPU= 17.094
  11:21:46 STEP 2648 H=2648:00 +CPU= 17.462
  11:22:13 STEP 2649 H=2649:00 +CPU= 26.668
  11:22:30 STEP 2650 H=2650:00 +CPU= 16.923
  11:22:47 STEP 2651 H=2651:00 +CPU= 16.572
  11:23:15 STEP 2652 H=2652:00 +CPU= 28.671
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002652
  11:23:32 STEP 2653 H=2653:00 +CPU= 16.546
  11:23:49 STEP 2654 H=2654:00 +CPU= 16.905
  11:24:15 STEP 2655 H=2655:00 +CPU= 26.333
  11:24:32 STEP 2656 H=2656:00 +CPU= 17.150
  11:24:49 STEP 2657 H=2657:00 +CPU= 16.465
  11:25:15 STEP 2658 H=2658:00 +CPU= 26.740
  11:25:32 STEP 2659 H=2659:00 +CPU= 16.839
  11:25:49 STEP 2660 H=2660:00 +CPU= 17.056
  11:26:15 STEP 2661 H=2661:00 +CPU= 25.615
  11:26:32 STEP 2662 H=2662:00 +CPU= 16.828
  11:26:48 STEP 2663 H=2663:00 +CPU= 16.472
  11:27:19 STEP 2664 H=2664:00 +CPU= 30.367
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_110.zip
Uploading the intermediate file: upload_file_110.zip
  11:27:35 STEP 2665 H=2665:00 +CPU= 16.290
  11:27:52 STEP 2666 H=2666:00 +CPU= 16.857
  11:28:18 STEP 2667 H=2667:00 +CPU= 26.526
  11:28:35 STEP 2668 H=2668:00 +CPU= 17.227
  11:28:52 STEP 2669 H=2669:00 +CPU= 16.136
  11:29:18 STEP 2670 H=2670:00 +CPU= 26.437
  11:29:35 STEP 2671 H=2671:00 +CPU= 16.469
  11:29:51 STEP 2672 H=2672:00 +CPU= 16.465
  11:30:16 STEP 2673 H=2673:00 +CPU= 25.187
  11:30:33 STEP 2674 H=2674:00 +CPU= 16.409
  11:30:49 STEP 2675 H=2675:00 +CPU= 16.004
  11:31:17 STEP 2676 H=2676:00 +CPU= 28.326
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002676
  11:31:33 STEP 2677 H=2677:00 +CPU= 16.337
  11:31:50 STEP 2678 H=2678:00 +CPU= 16.545
  11:32:16 STEP 2679 H=2679:00 +CPU= 25.738
  11:32:32 STEP 2680 H=2680:00 +CPU= 16.903
  11:32:49 STEP 2681 H=2681:00 +CPU= 16.578
  11:33:15 STEP 2682 H=2682:00 +CPU= 25.796
  11:33:31 STEP 2683 H=2683:00 +CPU= 16.189
  11:33:47 STEP 2684 H=2684:00 +CPU= 16.064
  11:34:12 STEP 2685 H=2685:00 +CPU= 25.014
  11:34:29 STEP 2686 H=2686:00 +CPU= 16.905
  11:34:46 STEP 2687 H=2687:00 +CPU= 16.905
  11:35:18 STEP 2688 H=2688:00 +CPU= 31.741
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_111.zip
Uploading the intermediate file: upload_file_111.zip
  11:35:35 STEP 2689 H=2689:00 +CPU= 17.319
  11:35:52 STEP 2690 H=2690:00 +CPU= 17.019
  11:36:18 STEP 2691 H=2691:00 +CPU= 26.410
  11:36:36 STEP 2692 H=2692:00 +CPU= 17.169
  11:36:52 STEP 2693 H=2693:00 +CPU= 16.217
  11:37:17 STEP 2694 H=2694:00 +CPU= 25.231
  11:37:33 STEP 2695 H=2695:00 +CPU= 15.832
  11:37:50 STEP 2696 H=2696:00 +CPU= 16.595
  11:38:16 STEP 2697 H=2697:00 +CPU= 25.966
  11:38:33 STEP 2698 H=2698:00 +CPU= 17.196
  11:38:49 STEP 2699 H=2699:00 +CPU= 16.273
  11:39:19 STEP 2700 H=2700:00 +CPU= 29.930
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002700
  11:39:35 STEP 2701 H=2701:00 +CPU= 16.315
  11:39:52 STEP 2702 H=2702:00 +CPU= 16.421
  11:40:16 STEP 2703 H=2703:00 +CPU= 24.706
  11:40:33 STEP 2704 H=2704:00 +CPU= 16.245
  11:40:48 STEP 2705 H=2705:00 +CPU= 15.295
  11:41:13 STEP 2706 H=2706:00 +CPU= 24.603
  11:41:28 STEP 2707 H=2707:00 +CPU= 15.403
  11:41:44 STEP 2708 H=2708:00 +CPU= 15.855
  11:42:08 STEP 2709 H=2709:00 +CPU= 23.927
  11:42:23 STEP 2710 H=2710:00 +CPU= 15.704
  11:42:39 STEP 2711 H=2711:00 +CPU= 15.833
  11:43:08 STEP 2712 H=2712:00 +CPU= 28.410
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_112.zip
Uploading the intermediate file: upload_file_112.zip
  11:43:23 STEP 2713 H=2713:00 +CPU= 15.481
  11:43:38 STEP 2714 H=2714:00 +CPU= 15.348
  11:44:02 STEP 2715 H=2715:00 +CPU= 23.935
  11:44:17 STEP 2716 H=2716:00 +CPU= 14.706
  11:44:31 STEP 2717 H=2717:00 +CPU= 14.149
  11:44:55 STEP 2718 H=2718:00 +CPU= 23.737
  11:45:10 STEP 2719 H=2719:00 +CPU= 14.681
  11:45:25 STEP 2720 H=2720:00 +CPU= 15.206
  11:45:48 STEP 2721 H=2721:00 +CPU= 23.184
  11:46:03 STEP 2722 H=2722:00 +CPU= 14.954
  11:46:19 STEP 2723 H=2723:00 +CPU= 15.634
  11:46:44 STEP 2724 H=2724:00 +CPU= 25.547
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002724
  11:46:58 STEP 2725 H=2725:00 +CPU= 14.210
  11:47:13 STEP 2726 H=2726:00 +CPU= 14.554
  11:47:35 STEP 2727 H=2727:00 +CPU= 22.420
  11:47:50 STEP 2728 H=2728:00 +CPU= 14.340
  11:48:04 STEP 2729 H=2729:00 +CPU= 14.005
  11:48:26 STEP 2730 H=2730:00 +CPU= 22.245
  11:48:40 STEP 2731 H=2731:00 +CPU= 13.736
  11:48:54 STEP 2732 H=2732:00 +CPU= 14.061
  11:49:16 STEP 2733 H=2733:00 +CPU= 22.051
  11:49:30 STEP 2734 H=2734:00 +CPU= 14.182
  11:49:44 STEP 2735 H=2735:00 +CPU= 14.366
  11:50:10 STEP 2736 H=2736:00 +CPU= 25.545
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_113.zip
Uploading the intermediate file: upload_file_113.zip
  11:50:24 STEP 2737 H=2737:00 +CPU= 14.084
  11:50:38 STEP 2738 H=2738:00 +CPU= 14.105
  11:51:00 STEP 2739 H=2739:00 +CPU= 21.936
  11:51:14 STEP 2740 H=2740:00 +CPU= 14.249
  11:51:28 STEP 2741 H=2741:00 +CPU= 13.936
  11:51:51 STEP 2742 H=2742:00 +CPU= 22.628
  11:52:05 STEP 2743 H=2743:00 +CPU= 14.412
  11:52:19 STEP 2744 H=2744:00 +CPU= 13.996
  11:52:42 STEP 2745 H=2745:00 +CPU= 22.201
  11:52:56 STEP 2746 H=2746:00 +CPU= 14.270
  11:53:09 STEP 2747 H=2747:00 +CPU= 13.618
  11:53:34 STEP 2748 H=2748:00 +CPU= 24.443
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002748
  11:53:47 STEP 2749 H=2749:00 +CPU= 13.480
  11:54:01 STEP 2750 H=2750:00 +CPU= 13.995
  11:54:23 STEP 2751 H=2751:00 +CPU= 21.612
  11:54:37 STEP 2752 H=2752:00 +CPU= 14.011
  11:54:51 STEP 2753 H=2753:00 +CPU= 13.603
  11:55:12 STEP 2754 H=2754:00 +CPU= 21.727
  11:55:26 STEP 2755 H=2755:00 +CPU= 13.288
  11:55:39 STEP 2756 H=2756:00 +CPU= 13.286
  11:55:59 STEP 2757 H=2757:00 +CPU= 20.594
  11:56:12 STEP 2758 H=2758:00 +CPU= 12.783
  11:56:23 STEP 2759 H=2759:00 +CPU= 11.006
  11:56:40 STEP 2760 H=2760:00 +CPU= 16.338
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_114.zip
Uploading the intermediate file: upload_file_114.zip
  11:56:48 STEP 2761 H=2761:00 +CPU=  8.521
  11:56:57 STEP 2762 H=2762:00 +CPU=  8.786
  11:57:11 STEP 2763 H=2763:00 +CPU= 13.818
  11:57:20 STEP 2764 H=2764:00 +CPU=  8.842
  11:57:29 STEP 2765 H=2765:00 +CPU=  9.669
  11:57:43 STEP 2766 H=2766:00 +CPU= 13.942
  11:57:52 STEP 2767 H=2767:00 +CPU=  8.435
  11:58:00 STEP 2768 H=2768:00 +CPU=  8.722
  11:58:14 STEP 2769 H=2769:00 +CPU= 13.622
  11:58:23 STEP 2770 H=2770:00 +CPU=  8.745
  11:58:31 STEP 2771 H=2771:00 +CPU=  8.496
  11:58:46 STEP 2772 H=2772:00 +CPU= 15.240
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002772
  11:58:55 STEP 2773 H=2773:00 +CPU=  8.525
  11:59:04 STEP 2774 H=2774:00 +CPU=  8.683
  11:59:17 STEP 2775 H=2775:00 +CPU= 13.709
  11:59:26 STEP 2776 H=2776:00 +CPU=  8.780
  11:59:35 STEP 2777 H=2777:00 +CPU=  8.548
  11:59:49 STEP 2778 H=2778:00 +CPU= 14.142
  11:59:57 STEP 2779 H=2779:00 +CPU=  8.550
  12:00:06 STEP 2780 H=2780:00 +CPU=  8.830
  12:00:20 STEP 2781 H=2781:00 +CPU= 13.637
  12:00:29 STEP 2782 H=2782:00 +CPU=  8.767
  12:00:37 STEP 2783 H=2783:00 +CPU=  8.690
  12:00:54 STEP 2784 H=2784:00 +CPU= 16.462
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_115.zip
Uploading the intermediate file: upload_file_115.zip
  12:01:03 STEP 2785 H=2785:00 +CPU=  8.840
  12:01:12 STEP 2786 H=2786:00 +CPU=  8.742
Uploading trickle at timestep: 10022400
  12:01:25 STEP 2787 H=2787:00 +CPU= 13.884
  12:01:34 STEP 2788 H=2788:00 +CPU=  8.821
  12:01:43 STEP 2789 H=2789:00 +CPU=  8.576
  12:01:57 STEP 2790 H=2790:00 +CPU= 13.986
  12:02:05 STEP 2791 H=2791:00 +CPU=  8.552
  12:02:14 STEP 2792 H=2792:00 +CPU=  8.884
  12:02:28 STEP 2793 H=2793:00 +CPU= 13.843
  12:02:37 STEP 2794 H=2794:00 +CPU=  8.920
  12:02:45 STEP 2795 H=2795:00 +CPU=  8.454
  12:03:01 STEP 2796 H=2796:00 +CPU= 15.538
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002796
  12:03:10 STEP 2797 H=2797:00 +CPU=  8.738
  12:03:19 STEP 2798 H=2798:00 +CPU=  8.880
  12:03:34 STEP 2799 H=2799:00 +CPU= 15.032
  12:03:43 STEP 2800 H=2800:00 +CPU=  9.225
  12:03:51 STEP 2801 H=2801:00 +CPU=  8.616
  12:04:05 STEP 2802 H=2802:00 +CPU= 13.981
  12:04:14 STEP 2803 H=2803:00 +CPU=  8.485
  12:04:22 STEP 2804 H=2804:00 +CPU=  8.480
  12:04:36 STEP 2805 H=2805:00 +CPU= 13.424
  12:04:44 STEP 2806 H=2806:00 +CPU=  8.563
  12:04:53 STEP 2807 H=2807:00 +CPU=  8.493
  12:05:09 STEP 2808 H=2808:00 +CPU= 15.656
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_116.zip
Uploading the intermediate file: upload_file_116.zip
  12:05:17 STEP 2809 H=2809:00 +CPU=  8.215
  12:05:25 STEP 2810 H=2810:00 +CPU=  8.444
  12:05:39 STEP 2811 H=2811:00 +CPU= 13.323
  12:05:47 STEP 2812 H=2812:00 +CPU=  8.413
  12:05:55 STEP 2813 H=2813:00 +CPU=  8.237
  12:06:09 STEP 2814 H=2814:00 +CPU= 13.533
  12:06:17 STEP 2815 H=2815:00 +CPU=  8.276
  12:06:26 STEP 2816 H=2816:00 +CPU=  8.559
  12:06:39 STEP 2817 H=2817:00 +CPU= 13.368
  12:06:47 STEP 2818 H=2818:00 +CPU=  8.443
  12:06:56 STEP 2819 H=2819:00 +CPU=  8.226
  12:07:11 STEP 2820 H=2820:00 +CPU= 14.863
  12:07:19 STEP 2821 H=2821:00 +CPU=  8.202
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002820
  12:07:27 STEP 2822 H=2822:00 +CPU=  8.542
  12:07:42 STEP 2823 H=2823:00 +CPU= 14.924
  12:07:51 STEP 2824 H=2824:00 +CPU=  8.687
  12:08:00 STEP 2825 H=2825:00 +CPU=  8.587
  12:08:15 STEP 2826 H=2826:00 +CPU= 15.342
  12:08:24 STEP 2827 H=2827:00 +CPU=  8.709
  12:08:32 STEP 2828 H=2828:00 +CPU=  8.856
  12:08:46 STEP 2829 H=2829:00 +CPU= 13.568
  12:08:55 STEP 2830 H=2830:00 +CPU=  8.724
  12:09:03 STEP 2831 H=2831:00 +CPU=  8.707
  12:09:20 STEP 2832 H=2832:00 +CPU= 16.201
  12:09:28 STEP 2833 H=2833:00 +CPU=  8.680
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_117.zip
Uploading the intermediate file: upload_file_117.zip
  12:09:37 STEP 2834 H=2834:00 +CPU=  8.802
  12:09:51 STEP 2835 H=2835:00 +CPU= 13.679
  12:10:00 STEP 2836 H=2836:00 +CPU=  9.391
  12:10:09 STEP 2837 H=2837:00 +CPU=  8.676
  12:10:23 STEP 2838 H=2838:00 +CPU= 13.889
  12:10:31 STEP 2839 H=2839:00 +CPU=  8.529
  12:10:40 STEP 2840 H=2840:00 +CPU=  8.809
  12:10:54 STEP 2841 H=2841:00 +CPU= 13.901
  12:11:03 STEP 2842 H=2842:00 +CPU=  8.786
  12:11:11 STEP 2843 H=2843:00 +CPU=  8.625
  12:11:27 STEP 2844 H=2844:00 +CPU= 15.332
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002844
  12:11:35 STEP 2845 H=2845:00 +CPU=  8.610
  12:11:44 STEP 2846 H=2846:00 +CPU=  9.027
  12:11:58 STEP 2847 H=2847:00 +CPU= 13.754
  12:12:07 STEP 2848 H=2848:00 +CPU=  8.795
  12:12:16 STEP 2849 H=2849:00 +CPU=  8.766
  12:12:30 STEP 2850 H=2850:00 +CPU= 14.454
  12:12:39 STEP 2851 H=2851:00 +CPU=  8.851
  12:12:48 STEP 2852 H=2852:00 +CPU=  8.741
  12:13:01 STEP 2853 H=2853:00 +CPU= 13.713
  12:13:10 STEP 2854 H=2854:00 +CPU=  8.797
  12:13:19 STEP 2855 H=2855:00 +CPU=  8.730
  12:13:35 STEP 2856 H=2856:00 +CPU= 16.425
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_118.zip
Uploading the intermediate file: upload_file_118.zip
  12:13:44 STEP 2857 H=2857:00 +CPU=  8.630
  12:13:53 STEP 2858 H=2858:00 +CPU=  8.783
  12:14:07 STEP 2859 H=2859:00 +CPU= 13.901
  12:14:18 STEP 2860 H=2860:00 +CPU= 11.474
  12:14:32 STEP 2861 H=2861:00 +CPU= 14.011
  12:14:55 STEP 2862 H=2862:00 +CPU= 22.365
  12:15:08 STEP 2863 H=2863:00 +CPU= 13.590
  12:15:23 STEP 2864 H=2864:00 +CPU= 14.595
  12:15:51 STEP 2865 H=2865:00 +CPU= 27.741
  12:16:07 STEP 2866 H=2866:00 +CPU= 16.467
  12:16:26 STEP 2867 H=2867:00 +CPU= 19.429
  12:17:00 STEP 2868 H=2868:00 +CPU= 33.567
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002868
  12:17:18 STEP 2869 H=2869:00 +CPU= 18.432
  12:17:37 STEP 2870 H=2870:00 +CPU= 18.735
  12:18:03 STEP 2871 H=2871:00 +CPU= 26.265
  12:18:20 STEP 2872 H=2872:00 +CPU= 16.995
  12:18:37 STEP 2873 H=2873:00 +CPU= 16.923
  12:19:05 STEP 2874 H=2874:00 +CPU= 27.757
  12:19:22 STEP 2875 H=2875:00 +CPU= 17.197
  12:19:41 STEP 2876 H=2876:00 +CPU= 19.158
  12:20:10 STEP 2877 H=2877:00 +CPU= 28.024
  12:20:28 STEP 2878 H=2878:00 +CPU= 18.176
  12:20:46 STEP 2879 H=2879:00 +CPU= 17.871
  12:21:20 STEP 2880 H=2880:00 +CPU= 34.487
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_119.zip
Uploading the intermediate file: upload_file_119.zip
  12:21:36 STEP 2881 H=2881:00 +CPU= 16.105
  12:21:54 STEP 2882 H=2882:00 +CPU= 17.422
  12:22:21 STEP 2883 H=2883:00 +CPU= 27.189
  12:22:37 STEP 2884 H=2884:00 +CPU= 16.444
  12:22:55 STEP 2885 H=2885:00 +CPU= 17.833
  12:23:22 STEP 2886 H=2886:00 +CPU= 27.075
  12:23:39 STEP 2887 H=2887:00 +CPU= 16.581
  12:23:57 STEP 2888 H=2888:00 +CPU= 17.802
  12:24:23 STEP 2889 H=2889:00 +CPU= 26.403
  12:24:40 STEP 2890 H=2890:00 +CPU= 17.169
  12:24:57 STEP 2891 H=2891:00 +CPU= 17.064
  12:25:25 STEP 2892 H=2892:00 +CPU= 28.125
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002892
  12:25:41 STEP 2893 H=2893:00 +CPU= 15.350
  12:25:56 STEP 2894 H=2894:00 +CPU= 15.730
  12:26:19 STEP 2895 H=2895:00 +CPU= 22.525
  12:26:33 STEP 2896 H=2896:00 +CPU= 14.023
  12:26:46 STEP 2897 H=2897:00 +CPU= 12.732
  12:27:07 STEP 2898 H=2898:00 +CPU= 21.016
  12:27:19 STEP 2899 H=2899:00 +CPU= 12.371
  12:27:32 STEP 2900 H=2900:00 +CPU= 12.662
  12:27:51 STEP 2901 H=2901:00 +CPU= 19.688
  12:28:05 STEP 2902 H=2902:00 +CPU= 13.382
  12:28:17 STEP 2903 H=2903:00 +CPU= 12.440
  12:28:40 STEP 2904 H=2904:00 +CPU= 23.030
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_120.zip
Uploading the intermediate file: upload_file_120.zip
  12:28:52 STEP 2905 H=2905:00 +CPU= 11.884
  12:29:04 STEP 2906 H=2906:00 +CPU= 12.140
  12:29:23 STEP 2907 H=2907:00 +CPU= 19.110
  12:29:35 STEP 2908 H=2908:00 +CPU= 12.045
  12:29:47 STEP 2909 H=2909:00 +CPU= 11.452
  12:30:05 STEP 2910 H=2910:00 +CPU= 18.376
  12:30:17 STEP 2911 H=2911:00 +CPU= 11.888
  12:30:30 STEP 2912 H=2912:00 +CPU= 12.388
  12:30:50 STEP 2913 H=2913:00 +CPU= 19.934
  12:31:02 STEP 2914 H=2914:00 +CPU= 12.526
  12:31:15 STEP 2915 H=2915:00 +CPU= 12.455
  12:31:37 STEP 2916 H=2916:00 +CPU= 22.530
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002916
  12:31:50 STEP 2917 H=2917:00 +CPU= 12.522
  12:32:02 STEP 2918 H=2918:00 +CPU= 12.680
  12:32:22 STEP 2919 H=2919:00 +CPU= 19.537
  12:32:35 STEP 2920 H=2920:00 +CPU= 12.793
  12:32:47 STEP 2921 H=2921:00 +CPU= 12.313
  12:33:07 STEP 2922 H=2922:00 +CPU= 20.077
  12:33:19 STEP 2923 H=2923:00 +CPU= 12.051
  12:33:31 STEP 2924 H=2924:00 +CPU= 11.788
  12:33:49 STEP 2925 H=2925:00 +CPU= 18.539
  12:34:01 STEP 2926 H=2926:00 +CPU= 11.823
  12:34:13 STEP 2927 H=2927:00 +CPU= 11.830
  12:34:34 STEP 2928 H=2928:00 +CPU= 21.483
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_121.zip
Uploading the intermediate file: upload_file_121.zip
  12:34:46 STEP 2929 H=2929:00 +CPU= 11.313
  12:34:57 STEP 2930 H=2930:00 +CPU= 11.679
  12:35:16 STEP 2931 H=2931:00 +CPU= 18.246
  12:35:28 STEP 2932 H=2932:00 +CPU= 12.574
  12:35:40 STEP 2933 H=2933:00 +CPU= 11.825
  12:36:00 STEP 2934 H=2934:00 +CPU= 20.327
  12:36:13 STEP 2935 H=2935:00 +CPU= 12.370
  12:36:25 STEP 2936 H=2936:00 +CPU= 12.605
  12:36:45 STEP 2937 H=2937:00 +CPU= 19.539
  12:36:58 STEP 2938 H=2938:00 +CPU= 12.622
  12:37:10 STEP 2939 H=2939:00 +CPU= 12.167
  12:37:32 STEP 2940 H=2940:00 +CPU= 21.802
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002940
  12:37:43 STEP 2941 H=2941:00 +CPU= 11.691
  12:37:56 STEP 2942 H=2942:00 +CPU= 12.553
  12:38:15 STEP 2943 H=2943:00 +CPU= 19.273
  12:38:28 STEP 2944 H=2944:00 +CPU= 12.516
  12:38:40 STEP 2945 H=2945:00 +CPU= 11.974
  12:39:00 STEP 2946 H=2946:00 +CPU= 19.939
  12:39:12 STEP 2947 H=2947:00 +CPU= 12.016
  12:39:24 STEP 2948 H=2948:00 +CPU= 12.203
  12:39:43 STEP 2949 H=2949:00 +CPU= 18.911
  12:39:55 STEP 2950 H=2950:00 +CPU= 12.233
  12:40:07 STEP 2951 H=2951:00 +CPU= 12.309
  12:40:30 STEP 2952 H=2952:00 +CPU= 22.458
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12196774/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0130_2011050100_123_980_12196774_1_r1247952588_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
12:43:53 (73): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:32:10 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 10,623,600 60,868 0.0057
09 May 2023 16:32:10 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 10,022,400 58,496 0.0058
09 May 2023 16:31:58 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 9,158,400 54,167 0.0059
09 May 2023 16:31:52 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 8,294,400 49,340 0.0059
09 May 2023 16:31:46 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 7,430,400 44,186 0.0059
09 May 2023 16:31:40 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 6,566,400 40,841 0.0062
09 May 2023 16:31:28 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 5,702,400 33,717 0.0059
09 May 2023 16:31:14 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 4,838,400 26,569 0.0055
09 May 2023 16:31:04 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 3,974,400 19,770 0.0050
09 May 2023 16:30:43 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 2,592,000 9,024 0.0035
09 May 2023 16:30:37 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 1,728,000 5,989 0.0035
09 May 2023 16:30:37 1537625 22300072 oifs_43r3_ps_0130_2011050100_123_980_12196774_1 864,000 3,015 0.0035


©2024 climateprediction.net