climateprediction.net home page
Task 22261394

Task 22261394

Name oifs_43r3_ps_0991_1990050100_123_959_12176635_0
Workunit 12176635
Created 21 Dec 2022, 20:37:51 UTC
Sent 16 Jan 2023, 20:47:31 UTC
Report deadline 17 Mar 2023, 20:47:31 UTC
Received 17 Jan 2023, 14:15:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1537625
Run time 10 hours 23 min 31 sec
CPU time 10 hours 14 min 31 sec
Validate state Invalid
Credit 1,209.60
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,795.41 MB
Peak swap size 4,974.26 MB
Peak disk usage 2,731.77 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Process still present 5 min after writing finish file; aborting</message>
<stderr_txt>
/ICMSHhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_94.zip
Uploading the intermediate file: upload_file_94.zip
  21:46:20 STEP 2281 H=2281:00 +CPU= 12.979
  21:46:32 STEP 2282 H=2282:00 +CPU= 12.091
  21:46:52 STEP 2283 H=2283:00 +CPU= 19.389
  21:47:05 STEP 2284 H=2284:00 +CPU= 13.577
  21:47:19 STEP 2285 H=2285:00 +CPU= 13.078
  21:47:39 STEP 2286 H=2286:00 +CPU= 20.855
  21:47:51 STEP 2287 H=2287:00 +CPU= 11.367
  21:48:01 STEP 2288 H=2288:00 +CPU= 10.069
  21:48:17 STEP 2289 H=2289:00 +CPU= 16.071
  21:48:28 STEP 2290 H=2290:00 +CPU= 11.024
  21:48:41 STEP 2291 H=2291:00 +CPU= 12.690
  21:49:04 STEP 2292 H=2292:00 +CPU= 23.634
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002292
  21:49:17 STEP 2293 H=2293:00 +CPU= 12.467
  21:49:29 STEP 2294 H=2294:00 +CPU= 12.460
  21:49:49 STEP 2295 H=2295:00 +CPU= 19.519
  21:50:01 STEP 2296 H=2296:00 +CPU= 12.571
  21:50:14 STEP 2297 H=2297:00 +CPU= 12.321
  21:50:35 STEP 2298 H=2298:00 +CPU= 20.957
  21:50:47 STEP 2299 H=2299:00 +CPU= 12.184
  21:51:00 STEP 2300 H=2300:00 +CPU= 12.780
  21:51:19 STEP 2301 H=2301:00 +CPU= 19.243
  21:51:32 STEP 2302 H=2302:00 +CPU= 13.326
  21:51:45 STEP 2303 H=2303:00 +CPU= 12.545
  21:52:08 STEP 2304 H=2304:00 +CPU= 23.604
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_95.zip
Uploading the intermediate file: upload_file_95.zip
  21:52:21 STEP 2305 H=2305:00 +CPU= 12.582
  21:52:34 STEP 2306 H=2306:00 +CPU= 12.737
  21:52:54 STEP 2307 H=2307:00 +CPU= 19.971
  21:53:07 STEP 2308 H=2308:00 +CPU= 13.573
  21:53:20 STEP 2309 H=2309:00 +CPU= 12.418
  21:53:42 STEP 2310 H=2310:00 +CPU= 21.999
  21:53:54 STEP 2311 H=2311:00 +CPU= 12.599
  21:54:07 STEP 2312 H=2312:00 +CPU= 12.185
  21:54:26 STEP 2313 H=2313:00 +CPU= 19.507
  21:54:39 STEP 2314 H=2314:00 +CPU= 13.142
  21:54:52 STEP 2315 H=2315:00 +CPU= 12.483
  21:55:15 STEP 2316 H=2316:00 +CPU= 22.958
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002316
  21:55:27 STEP 2317 H=2317:00 +CPU= 12.473
  21:55:40 STEP 2318 H=2318:00 +CPU= 12.831
  21:56:00 STEP 2319 H=2319:00 +CPU= 20.248
  21:56:13 STEP 2320 H=2320:00 +CPU= 12.549
  21:56:26 STEP 2321 H=2321:00 +CPU= 13.526
  21:56:48 STEP 2322 H=2322:00 +CPU= 21.509
  21:57:01 STEP 2323 H=2323:00 +CPU= 13.412
  21:57:15 STEP 2324 H=2324:00 +CPU= 13.116
  21:57:35 STEP 2325 H=2325:00 +CPU= 20.234
  21:57:47 STEP 2326 H=2326:00 +CPU= 12.502
  21:58:00 STEP 2327 H=2327:00 +CPU= 13.205
  21:58:24 STEP 2328 H=2328:00 +CPU= 23.241
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_96.zip
Uploading the intermediate file: upload_file_96.zip
  21:58:36 STEP 2329 H=2329:00 +CPU= 12.057
  21:58:49 STEP 2330 H=2330:00 +CPU= 12.795
  21:59:09 STEP 2331 H=2331:00 +CPU= 20.802
  21:59:22 STEP 2332 H=2332:00 +CPU= 12.719
  21:59:35 STEP 2333 H=2333:00 +CPU= 12.607
  21:59:55 STEP 2334 H=2334:00 +CPU= 19.871
  22:00:07 STEP 2335 H=2335:00 +CPU= 12.229
  22:00:20 STEP 2336 H=2336:00 +CPU= 12.851
  22:00:39 STEP 2337 H=2337:00 +CPU= 19.336
  22:00:52 STEP 2338 H=2338:00 +CPU= 12.909
  22:01:04 STEP 2339 H=2339:00 +CPU= 11.963
  22:01:27 STEP 2340 H=2340:00 +CPU= 22.457
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002340
  22:01:39 STEP 2341 H=2341:00 +CPU= 12.848
  22:01:52 STEP 2342 H=2342:00 +CPU= 13.038
  22:02:14 STEP 2343 H=2343:00 +CPU= 21.165
  22:02:26 STEP 2344 H=2344:00 +CPU= 12.805
  22:02:39 STEP 2345 H=2345:00 +CPU= 12.736
  22:03:01 STEP 2346 H=2346:00 +CPU= 21.327
  22:03:13 STEP 2347 H=2347:00 +CPU= 12.197
  22:03:27 STEP 2348 H=2348:00 +CPU= 13.730
  22:03:46 STEP 2349 H=2349:00 +CPU= 19.731
  22:03:59 STEP 2350 H=2350:00 +CPU= 12.741
  22:04:11 STEP 2351 H=2351:00 +CPU= 12.369
  22:04:34 STEP 2352 H=2352:00 +CPU= 23.074
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_97.zip
Uploading the intermediate file: upload_file_97.zip
  22:04:46 STEP 2353 H=2353:00 +CPU= 11.914
  22:05:00 STEP 2354 H=2354:00 +CPU= 13.641
  22:05:20 STEP 2355 H=2355:00 +CPU= 19.563
  22:05:32 STEP 2356 H=2356:00 +CPU= 12.352
  22:05:44 STEP 2357 H=2357:00 +CPU= 12.168
  22:06:04 STEP 2358 H=2358:00 +CPU= 19.893
  22:06:17 STEP 2359 H=2359:00 +CPU= 13.084
  22:06:30 STEP 2360 H=2360:00 +CPU= 12.721
  22:06:51 STEP 2361 H=2361:00 +CPU= 21.099
  22:07:04 STEP 2362 H=2362:00 +CPU= 12.897
  22:07:16 STEP 2363 H=2363:00 +CPU= 11.937
  22:07:39 STEP 2364 H=2364:00 +CPU= 22.934
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002364
  22:07:51 STEP 2365 H=2365:00 +CPU= 11.812
  22:08:04 STEP 2366 H=2366:00 +CPU= 12.877
  22:08:24 STEP 2367 H=2367:00 +CPU= 20.011
  22:08:37 STEP 2368 H=2368:00 +CPU= 13.445
  22:08:50 STEP 2369 H=2369:00 +CPU= 12.850
  22:09:10 STEP 2370 H=2370:00 +CPU= 20.201
  22:09:22 STEP 2371 H=2371:00 +CPU= 12.132
  22:09:35 STEP 2372 H=2372:00 +CPU= 13.124
  22:09:55 STEP 2373 H=2373:00 +CPU= 20.017
  22:10:08 STEP 2374 H=2374:00 +CPU= 12.460
  22:10:21 STEP 2375 H=2375:00 +CPU= 12.709
  22:10:45 STEP 2376 H=2376:00 +CPU= 24.144
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_98.zip
Uploading the intermediate file: upload_file_98.zip
  22:10:58 STEP 2377 H=2377:00 +CPU= 12.779
  22:11:11 STEP 2378 H=2378:00 +CPU= 13.631
  22:11:31 STEP 2379 H=2379:00 +CPU= 19.572
  22:11:44 STEP 2380 H=2380:00 +CPU= 12.698
  22:11:56 STEP 2381 H=2381:00 +CPU= 12.977
  22:12:17 STEP 2382 H=2382:00 +CPU= 20.296
  22:12:30 STEP 2383 H=2383:00 +CPU= 12.942
  22:12:42 STEP 2384 H=2384:00 +CPU= 12.578
  22:13:03 STEP 2385 H=2385:00 +CPU= 20.393
  22:13:15 STEP 2386 H=2386:00 +CPU= 12.573
  22:13:28 STEP 2387 H=2387:00 +CPU= 12.865
  22:13:50 STEP 2388 H=2388:00 +CPU= 21.994
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002388
  22:14:03 STEP 2389 H=2389:00 +CPU= 12.942
  22:14:17 STEP 2390 H=2390:00 +CPU= 14.098
  22:14:37 STEP 2391 H=2391:00 +CPU= 19.713
  22:14:50 STEP 2392 H=2392:00 +CPU= 12.722
  22:15:02 STEP 2393 H=2393:00 +CPU= 12.474
  22:15:22 STEP 2394 H=2394:00 +CPU= 19.704
  22:15:35 STEP 2395 H=2395:00 +CPU= 13.208
  22:15:48 STEP 2396 H=2396:00 +CPU= 12.745
  22:16:08 STEP 2397 H=2397:00 +CPU= 19.793
  22:16:21 STEP 2398 H=2398:00 +CPU= 13.125
  22:16:35 STEP 2399 H=2399:00 +CPU= 13.772
  22:16:58 STEP 2400 H=2400:00 +CPU= 23.291
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_99.zip
Uploading the intermediate file: upload_file_99.zip
  22:17:11 STEP 2401 H=2401:00 +CPU= 12.718
  22:17:23 STEP 2402 H=2402:00 +CPU= 12.633
Uploading trickle at timestep: 8640000
  22:17:43 STEP 2403 H=2403:00 +CPU= 19.878
  22:17:56 STEP 2404 H=2404:00 +CPU= 12.339
  22:18:09 STEP 2405 H=2405:00 +CPU= 13.183
  22:18:29 STEP 2406 H=2406:00 +CPU= 20.027
  22:18:42 STEP 2407 H=2407:00 +CPU= 13.140
  22:18:55 STEP 2408 H=2408:00 +CPU= 12.654
  22:19:14 STEP 2409 H=2409:00 +CPU= 19.784
  22:19:28 STEP 2410 H=2410:00 +CPU= 13.990
  22:19:40 STEP 2411 H=2411:00 +CPU= 12.044
  22:20:03 STEP 2412 H=2412:00 +CPU= 22.757
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002412
  22:20:16 STEP 2413 H=2413:00 +CPU= 12.706
  22:20:28 STEP 2414 H=2414:00 +CPU= 12.246
  22:20:48 STEP 2415 H=2415:00 +CPU= 19.879
  22:21:01 STEP 2416 H=2416:00 +CPU= 12.670
  22:21:13 STEP 2417 H=2417:00 +CPU= 12.176
  22:21:35 STEP 2418 H=2418:00 +CPU= 21.719
  22:21:47 STEP 2419 H=2419:00 +CPU= 12.475
  22:22:01 STEP 2420 H=2420:00 +CPU= 13.689
  22:22:20 STEP 2421 H=2421:00 +CPU= 19.682
  22:22:34 STEP 2422 H=2422:00 +CPU= 13.178
  22:22:47 STEP 2423 H=2423:00 +CPU= 13.601
  22:23:11 STEP 2424 H=2424:00 +CPU= 23.895
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_100.zip
Uploading the intermediate file: upload_file_100.zip
  22:23:24 STEP 2425 H=2425:00 +CPU= 13.136
  22:23:37 STEP 2426 H=2426:00 +CPU= 12.997
  22:23:58 STEP 2427 H=2427:00 +CPU= 20.783
  22:24:11 STEP 2428 H=2428:00 +CPU= 12.940
  22:24:24 STEP 2429 H=2429:00 +CPU= 12.629
  22:24:45 STEP 2430 H=2430:00 +CPU= 20.798
  22:24:57 STEP 2431 H=2431:00 +CPU= 12.259
  22:25:10 STEP 2432 H=2432:00 +CPU= 12.901
  22:25:30 STEP 2433 H=2433:00 +CPU= 19.879
  22:25:42 STEP 2434 H=2434:00 +CPU= 12.836
  22:25:54 STEP 2435 H=2435:00 +CPU= 11.911
  22:26:18 STEP 2436 H=2436:00 +CPU= 23.254
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002436
  22:26:29 STEP 2437 H=2437:00 +CPU= 11.624
  22:26:42 STEP 2438 H=2438:00 +CPU= 12.598
  22:27:02 STEP 2439 H=2439:00 +CPU= 20.096
  22:27:15 STEP 2440 H=2440:00 +CPU= 12.938
  22:27:27 STEP 2441 H=2441:00 +CPU= 12.400
  22:27:47 STEP 2442 H=2442:00 +CPU= 19.613
  22:27:59 STEP 2443 H=2443:00 +CPU= 12.063
  22:28:11 STEP 2444 H=2444:00 +CPU= 12.169
  22:28:31 STEP 2445 H=2445:00 +CPU= 19.597
  22:28:43 STEP 2446 H=2446:00 +CPU= 12.451
  22:28:56 STEP 2447 H=2447:00 +CPU= 12.546
  22:29:21 STEP 2448 H=2448:00 +CPU= 24.626
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_101.zip
Uploading the intermediate file: upload_file_101.zip
  22:29:33 STEP 2449 H=2449:00 +CPU= 12.310
  22:29:46 STEP 2450 H=2450:00 +CPU= 12.724
  22:30:07 STEP 2451 H=2451:00 +CPU= 21.123
  22:30:20 STEP 2452 H=2452:00 +CPU= 12.820
  22:30:32 STEP 2453 H=2453:00 +CPU= 12.405
  22:30:52 STEP 2454 H=2454:00 +CPU= 19.897
  22:31:04 STEP 2455 H=2455:00 +CPU= 11.706
  22:31:16 STEP 2456 H=2456:00 +CPU= 12.484
  22:31:36 STEP 2457 H=2457:00 +CPU= 19.448
  22:31:49 STEP 2458 H=2458:00 +CPU= 13.084
  22:32:01 STEP 2459 H=2459:00 +CPU= 11.999
  22:32:23 STEP 2460 H=2460:00 +CPU= 22.359
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002460
  22:32:35 STEP 2461 H=2461:00 +CPU= 11.776
  22:32:48 STEP 2462 H=2462:00 +CPU= 12.902
  22:33:07 STEP 2463 H=2463:00 +CPU= 19.125
  22:33:20 STEP 2464 H=2464:00 +CPU= 13.435
  22:33:33 STEP 2465 H=2465:00 +CPU= 12.505
  22:33:53 STEP 2466 H=2466:00 +CPU= 19.703
  22:34:05 STEP 2467 H=2467:00 +CPU= 12.044
  22:34:17 STEP 2468 H=2468:00 +CPU= 12.675
  22:34:38 STEP 2469 H=2469:00 +CPU= 20.484
  22:34:51 STEP 2470 H=2470:00 +CPU= 12.830
  22:35:04 STEP 2471 H=2471:00 +CPU= 12.864
  22:35:27 STEP 2472 H=2472:00 +CPU= 23.336
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_102.zip
Uploading the intermediate file: upload_file_102.zip
  22:35:41 STEP 2473 H=2473:00 +CPU= 13.989
  22:35:53 STEP 2474 H=2474:00 +CPU= 12.504
  22:36:12 STEP 2475 H=2475:00 +CPU= 19.039
  22:36:25 STEP 2476 H=2476:00 +CPU= 12.545
  22:36:37 STEP 2477 H=2477:00 +CPU= 12.326
  22:36:57 STEP 2478 H=2478:00 +CPU= 19.596
  22:37:09 STEP 2479 H=2479:00 +CPU= 12.127
  22:37:21 STEP 2480 H=2480:00 +CPU= 12.338
  22:37:42 STEP 2481 H=2481:00 +CPU= 20.478
  22:37:54 STEP 2482 H=2482:00 +CPU= 12.497
  22:38:06 STEP 2483 H=2483:00 +CPU= 11.646
  22:38:29 STEP 2484 H=2484:00 +CPU= 23.119
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002484
  22:38:42 STEP 2485 H=2485:00 +CPU= 13.137
  22:38:56 STEP 2486 H=2486:00 +CPU= 13.453
  22:39:15 STEP 2487 H=2487:00 +CPU= 19.429
  22:39:28 STEP 2488 H=2488:00 +CPU= 12.492
  22:39:40 STEP 2489 H=2489:00 +CPU= 12.200
  22:40:00 STEP 2490 H=2490:00 +CPU= 19.947
  22:40:12 STEP 2491 H=2491:00 +CPU= 12.404
  22:40:25 STEP 2492 H=2492:00 +CPU= 12.473
  22:40:44 STEP 2493 H=2493:00 +CPU= 19.193
  22:40:58 STEP 2494 H=2494:00 +CPU= 13.810
  22:41:10 STEP 2495 H=2495:00 +CPU= 12.169
  22:41:34 STEP 2496 H=2496:00 +CPU= 23.802
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_103.zip
Uploading the intermediate file: upload_file_103.zip
  22:41:46 STEP 2497 H=2497:00 +CPU= 12.118
  22:42:00 STEP 2498 H=2498:00 +CPU= 13.557
  22:42:20 STEP 2499 H=2499:00 +CPU= 20.001
  22:42:33 STEP 2500 H=2500:00 +CPU= 13.269
  22:42:45 STEP 2501 H=2501:00 +CPU= 11.812
  22:43:05 STEP 2502 H=2502:00 +CPU= 20.083
  22:43:17 STEP 2503 H=2503:00 +CPU= 12.158
  22:43:30 STEP 2504 H=2504:00 +CPU= 12.912
  22:43:51 STEP 2505 H=2505:00 +CPU= 21.076
  22:44:04 STEP 2506 H=2506:00 +CPU= 12.587
  22:44:16 STEP 2507 H=2507:00 +CPU= 12.079
  22:44:37 STEP 2508 H=2508:00 +CPU= 21.725
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002508
  22:44:49 STEP 2509 H=2509:00 +CPU= 11.907
  22:45:02 STEP 2510 H=2510:00 +CPU= 12.532
  22:45:21 STEP 2511 H=2511:00 +CPU= 19.433
  22:45:35 STEP 2512 H=2512:00 +CPU= 13.211
  22:45:47 STEP 2513 H=2513:00 +CPU= 12.022
  22:46:07 STEP 2514 H=2514:00 +CPU= 20.175
  22:46:19 STEP 2515 H=2515:00 +CPU= 11.840
  22:46:32 STEP 2516 H=2516:00 +CPU= 13.233
  22:46:52 STEP 2517 H=2517:00 +CPU= 19.821
  22:47:05 STEP 2518 H=2518:00 +CPU= 12.950
  22:47:18 STEP 2519 H=2519:00 +CPU= 13.415
  22:47:41 STEP 2520 H=2520:00 +CPU= 22.735
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_104.zip
Uploading the intermediate file: upload_file_104.zip
  22:47:53 STEP 2521 H=2521:00 +CPU= 12.268
  22:48:05 STEP 2522 H=2522:00 +CPU= 12.248
  22:48:25 STEP 2523 H=2523:00 +CPU= 19.698
  22:48:37 STEP 2524 H=2524:00 +CPU= 12.249
  22:48:50 STEP 2525 H=2525:00 +CPU= 12.571
  22:49:10 STEP 2526 H=2526:00 +CPU= 20.581
  22:49:24 STEP 2527 H=2527:00 +CPU= 13.021
  22:49:36 STEP 2528 H=2528:00 +CPU= 12.657
  22:49:57 STEP 2529 H=2529:00 +CPU= 20.793
  22:50:10 STEP 2530 H=2530:00 +CPU= 12.539
  22:50:21 STEP 2531 H=2531:00 +CPU= 11.832
  22:50:43 STEP 2532 H=2532:00 +CPU= 22.016
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002532
  22:50:55 STEP 2533 H=2533:00 +CPU= 11.900
  22:51:08 STEP 2534 H=2534:00 +CPU= 12.993
  22:51:28 STEP 2535 H=2535:00 +CPU= 19.222
  22:51:40 STEP 2536 H=2536:00 +CPU= 12.503
  22:51:53 STEP 2537 H=2537:00 +CPU= 12.486
  22:52:13 STEP 2538 H=2538:00 +CPU= 20.250
  22:52:25 STEP 2539 H=2539:00 +CPU= 12.120
  22:52:37 STEP 2540 H=2540:00 +CPU= 12.171
  22:52:57 STEP 2541 H=2541:00 +CPU= 19.964
  22:53:10 STEP 2542 H=2542:00 +CPU= 12.506
  22:53:22 STEP 2543 H=2543:00 +CPU= 12.616
  22:53:46 STEP 2544 H=2544:00 +CPU= 23.286
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_105.zip
Uploading the intermediate file: upload_file_105.zip
  22:53:58 STEP 2545 H=2545:00 +CPU= 12.226
  22:54:10 STEP 2546 H=2546:00 +CPU= 12.175
  22:54:30 STEP 2547 H=2547:00 +CPU= 19.607
  22:54:42 STEP 2548 H=2548:00 +CPU= 12.724
  22:54:55 STEP 2549 H=2549:00 +CPU= 12.267
  22:55:15 STEP 2550 H=2550:00 +CPU= 20.102
  22:55:26 STEP 2551 H=2551:00 +CPU= 11.470
  22:55:39 STEP 2552 H=2552:00 +CPU= 12.901
  22:55:59 STEP 2553 H=2553:00 +CPU= 19.395
  22:56:11 STEP 2554 H=2554:00 +CPU= 12.515
  22:56:23 STEP 2555 H=2555:00 +CPU= 12.279
  22:56:46 STEP 2556 H=2556:00 +CPU= 22.334
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002556
  22:56:59 STEP 2557 H=2557:00 +CPU= 13.284
  22:57:13 STEP 2558 H=2558:00 +CPU= 13.557
  22:57:32 STEP 2559 H=2559:00 +CPU= 19.293
  22:57:45 STEP 2560 H=2560:00 +CPU= 12.846
  22:57:57 STEP 2561 H=2561:00 +CPU= 12.428
  22:58:17 STEP 2562 H=2562:00 +CPU= 19.897
  22:58:29 STEP 2563 H=2563:00 +CPU= 12.032
  22:58:41 STEP 2564 H=2564:00 +CPU= 12.190
  22:59:03 STEP 2565 H=2565:00 +CPU= 21.659
  22:59:16 STEP 2566 H=2566:00 +CPU= 13.196
  22:59:30 STEP 2567 H=2567:00 +CPU= 13.467
  22:59:53 STEP 2568 H=2568:00 +CPU= 23.174
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_106.zip
Uploading the intermediate file: upload_file_106.zip
  23:00:05 STEP 2569 H=2569:00 +CPU= 11.958
  23:00:17 STEP 2570 H=2570:00 +CPU= 12.216
  23:00:36 STEP 2571 H=2571:00 +CPU= 19.224
  23:00:49 STEP 2572 H=2572:00 +CPU= 12.247
  23:01:01 STEP 2573 H=2573:00 +CPU= 12.135
  23:01:21 STEP 2574 H=2574:00 +CPU= 19.898
  23:01:32 STEP 2575 H=2575:00 +CPU= 11.706
  23:01:45 STEP 2576 H=2576:00 +CPU= 12.605
  23:02:04 STEP 2577 H=2577:00 +CPU= 19.170
  23:02:17 STEP 2578 H=2578:00 +CPU= 13.236
  23:02:30 STEP 2579 H=2579:00 +CPU= 12.735
  23:02:52 STEP 2580 H=2580:00 +CPU= 21.650
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002580
  23:03:04 STEP 2581 H=2581:00 +CPU= 12.138
  23:03:16 STEP 2582 H=2582:00 +CPU= 12.392
  23:03:38 STEP 2583 H=2583:00 +CPU= 21.086
  23:03:50 STEP 2584 H=2584:00 +CPU= 12.341
  23:04:02 STEP 2585 H=2585:00 +CPU= 11.847
  23:04:22 STEP 2586 H=2586:00 +CPU= 20.167
  23:04:34 STEP 2587 H=2587:00 +CPU= 12.372
  23:04:47 STEP 2588 H=2588:00 +CPU= 12.878
  23:05:07 STEP 2589 H=2589:00 +CPU= 19.945
  23:05:20 STEP 2590 H=2590:00 +CPU= 12.674
  23:05:32 STEP 2591 H=2591:00 +CPU= 12.466
  23:05:56 STEP 2592 H=2592:00 +CPU= 23.637
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_107.zip
Uploading the intermediate file: upload_file_107.zip
  23:06:08 STEP 2593 H=2593:00 +CPU= 11.753
  23:06:21 STEP 2594 H=2594:00 +CPU= 12.888
  23:06:41 STEP 2595 H=2595:00 +CPU= 20.225
  23:06:55 STEP 2596 H=2596:00 +CPU= 14.057
  23:07:07 STEP 2597 H=2597:00 +CPU= 12.192
  23:07:27 STEP 2598 H=2598:00 +CPU= 19.990
  23:07:40 STEP 2599 H=2599:00 +CPU= 12.574
  23:07:53 STEP 2600 H=2600:00 +CPU= 13.387
  23:08:13 STEP 2601 H=2601:00 +CPU= 19.417
  23:08:25 STEP 2602 H=2602:00 +CPU= 12.041
  23:08:37 STEP 2603 H=2603:00 +CPU= 11.958
  23:08:59 STEP 2604 H=2604:00 +CPU= 22.191
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002604
  23:09:12 STEP 2605 H=2605:00 +CPU= 12.937
  23:09:25 STEP 2606 H=2606:00 +CPU= 12.927
  23:09:44 STEP 2607 H=2607:00 +CPU= 19.112
  23:09:56 STEP 2608 H=2608:00 +CPU= 12.242
  23:10:08 STEP 2609 H=2609:00 +CPU= 11.979
  23:10:28 STEP 2610 H=2610:00 +CPU= 19.839
  23:10:42 STEP 2611 H=2611:00 +CPU= 13.689
  23:10:54 STEP 2612 H=2612:00 +CPU= 12.389
  23:11:14 STEP 2613 H=2613:00 +CPU= 19.910
  23:11:26 STEP 2614 H=2614:00 +CPU= 12.536
  23:11:40 STEP 2615 H=2615:00 +CPU= 13.339
  23:12:03 STEP 2616 H=2616:00 +CPU= 23.633
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_108.zip
Uploading the intermediate file: upload_file_108.zip
  23:12:16 STEP 2617 H=2617:00 +CPU= 12.420
  23:12:28 STEP 2618 H=2618:00 +CPU= 12.320
  23:12:48 STEP 2619 H=2619:00 +CPU= 19.898
  23:13:01 STEP 2620 H=2620:00 +CPU= 13.264
  23:13:14 STEP 2621 H=2621:00 +CPU= 12.164
  23:13:36 STEP 2622 H=2622:00 +CPU= 22.291
  23:13:48 STEP 2623 H=2623:00 +CPU= 12.480
  23:14:01 STEP 2624 H=2624:00 +CPU= 12.990
  23:14:21 STEP 2625 H=2625:00 +CPU= 19.791
  23:14:34 STEP 2626 H=2626:00 +CPU= 13.086
  23:14:47 STEP 2627 H=2627:00 +CPU= 12.483
  23:15:09 STEP 2628 H=2628:00 +CPU= 22.326
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002628
  23:15:22 STEP 2629 H=2629:00 +CPU= 12.795
  23:15:35 STEP 2630 H=2630:00 +CPU= 13.066
  23:15:54 STEP 2631 H=2631:00 +CPU= 19.329
  23:16:08 STEP 2632 H=2632:00 +CPU= 13.812
  23:16:21 STEP 2633 H=2633:00 +CPU= 12.670
  23:16:40 STEP 2634 H=2634:00 +CPU= 19.483
  23:16:53 STEP 2635 H=2635:00 +CPU= 12.777
  23:17:07 STEP 2636 H=2636:00 +CPU= 13.373
  23:17:26 STEP 2637 H=2637:00 +CPU= 19.464
  23:17:38 STEP 2638 H=2638:00 +CPU= 12.314
  23:17:51 STEP 2639 H=2639:00 +CPU= 12.473
  23:18:14 STEP 2640 H=2640:00 +CPU= 23.299
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_109.zip
Uploading the intermediate file: upload_file_109.zip
  23:18:26 STEP 2641 H=2641:00 +CPU= 12.053
Uploading trickle at timestep: 9504000
  23:18:40 STEP 2642 H=2642:00 +CPU= 13.779
  23:19:00 STEP 2643 H=2643:00 +CPU= 19.800
  23:19:12 STEP 2644 H=2644:00 +CPU= 12.516
  23:19:24 STEP 2645 H=2645:00 +CPU= 12.017
  23:19:44 STEP 2646 H=2646:00 +CPU= 19.886
  23:19:56 STEP 2647 H=2647:00 +CPU= 11.995
  23:20:09 STEP 2648 H=2648:00 +CPU= 13.028
  23:20:30 STEP 2649 H=2649:00 +CPU= 20.387
  23:20:43 STEP 2650 H=2650:00 +CPU= 13.249
  23:20:55 STEP 2651 H=2651:00 +CPU= 12.296
  23:21:18 STEP 2652 H=2652:00 +CPU= 22.259
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002652
  23:21:31 STEP 2653 H=2653:00 +CPU= 13.347
  23:21:44 STEP 2654 H=2654:00 +CPU= 13.193
  23:22:04 STEP 2655 H=2655:00 +CPU= 19.408
  23:22:16 STEP 2656 H=2656:00 +CPU= 12.324
  23:22:28 STEP 2657 H=2657:00 +CPU= 12.282
  23:22:48 STEP 2658 H=2658:00 +CPU= 19.965
  23:23:00 STEP 2659 H=2659:00 +CPU= 12.011
  23:23:13 STEP 2660 H=2660:00 +CPU= 12.595
  23:23:33 STEP 2661 H=2661:00 +CPU= 20.317
  23:23:45 STEP 2662 H=2662:00 +CPU= 12.159
  23:23:58 STEP 2663 H=2663:00 +CPU= 12.767
  23:24:22 STEP 2664 H=2664:00 +CPU= 23.963
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_110.zip
Uploading the intermediate file: upload_file_110.zip
  23:24:35 STEP 2665 H=2665:00 +CPU= 12.536
  23:24:47 STEP 2666 H=2666:00 +CPU= 12.233
  23:25:06 STEP 2667 H=2667:00 +CPU= 19.692
  23:25:20 STEP 2668 H=2668:00 +CPU= 13.294
  23:25:33 STEP 2669 H=2669:00 +CPU= 13.403
  23:25:53 STEP 2670 H=2670:00 +CPU= 20.164
  23:26:06 STEP 2671 H=2671:00 +CPU= 12.706
  23:26:19 STEP 2672 H=2672:00 +CPU= 13.072
  23:26:39 STEP 2673 H=2673:00 +CPU= 20.321
  23:26:52 STEP 2674 H=2674:00 +CPU= 12.355
  23:27:04 STEP 2675 H=2675:00 +CPU= 12.120
  23:27:26 STEP 2676 H=2676:00 +CPU= 22.371
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002676
  23:27:38 STEP 2677 H=2677:00 +CPU= 11.853
  23:27:51 STEP 2678 H=2678:00 +CPU= 12.802
  23:28:11 STEP 2679 H=2679:00 +CPU= 20.438
  23:28:24 STEP 2680 H=2680:00 +CPU= 12.757
  23:28:37 STEP 2681 H=2681:00 +CPU= 12.565
  23:28:58 STEP 2682 H=2682:00 +CPU= 21.113
  23:29:10 STEP 2683 H=2683:00 +CPU= 11.718
  23:29:23 STEP 2684 H=2684:00 +CPU= 13.013
  23:29:43 STEP 2685 H=2685:00 +CPU= 20.742
  23:29:58 STEP 2686 H=2686:00 +CPU= 14.138
  23:30:11 STEP 2687 H=2687:00 +CPU= 13.430
  23:30:36 STEP 2688 H=2688:00 +CPU= 24.454
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_111.zip
Uploading the intermediate file: upload_file_111.zip
  23:30:49 STEP 2689 H=2689:00 +CPU= 13.979
  23:31:02 STEP 2690 H=2690:00 +CPU= 12.188
  23:31:22 STEP 2691 H=2691:00 +CPU= 19.911
  23:31:34 STEP 2692 H=2692:00 +CPU= 12.334
  23:31:47 STEP 2693 H=2693:00 +CPU= 12.996
  23:32:07 STEP 2694 H=2694:00 +CPU= 19.834
  23:32:19 STEP 2695 H=2695:00 +CPU= 12.177
  23:32:32 STEP 2696 H=2696:00 +CPU= 12.892
  23:32:52 STEP 2697 H=2697:00 +CPU= 19.978
  23:33:04 STEP 2698 H=2698:00 +CPU= 12.589
  23:33:17 STEP 2699 H=2699:00 +CPU= 12.433
  23:33:38 STEP 2700 H=2700:00 +CPU= 21.207
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002700
  23:33:51 STEP 2701 H=2701:00 +CPU= 12.885
  23:34:04 STEP 2702 H=2702:00 +CPU= 12.669
  23:34:24 STEP 2703 H=2703:00 +CPU= 19.980
  23:34:37 STEP 2704 H=2704:00 +CPU= 13.152
  23:34:49 STEP 2705 H=2705:00 +CPU= 11.683
  23:35:09 STEP 2706 H=2706:00 +CPU= 20.448
  23:35:21 STEP 2707 H=2707:00 +CPU= 12.057
  23:35:34 STEP 2708 H=2708:00 +CPU= 12.433
  23:35:53 STEP 2709 H=2709:00 +CPU= 19.411
  23:36:05 STEP 2710 H=2710:00 +CPU= 12.172
  23:36:19 STEP 2711 H=2711:00 +CPU= 13.549
  23:36:42 STEP 2712 H=2712:00 +CPU= 23.567
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_112.zip
Uploading the intermediate file: upload_file_112.zip
  23:36:55 STEP 2713 H=2713:00 +CPU= 13.096
  23:37:09 STEP 2714 H=2714:00 +CPU= 13.819
  23:37:30 STEP 2715 H=2715:00 +CPU= 20.349
  23:37:42 STEP 2716 H=2716:00 +CPU= 12.555
  23:37:55 STEP 2717 H=2717:00 +CPU= 12.361
  23:38:15 STEP 2718 H=2718:00 +CPU= 19.959
  23:38:26 STEP 2719 H=2719:00 +CPU= 11.870
  23:38:39 STEP 2720 H=2720:00 +CPU= 12.704
  23:39:00 STEP 2721 H=2721:00 +CPU= 20.534
  23:39:12 STEP 2722 H=2722:00 +CPU= 12.354
  23:39:24 STEP 2723 H=2723:00 +CPU= 12.088
  23:39:45 STEP 2724 H=2724:00 +CPU= 21.226
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002724
  23:39:58 STEP 2725 H=2725:00 +CPU= 12.329
  23:40:10 STEP 2726 H=2726:00 +CPU= 12.365
  23:40:29 STEP 2727 H=2727:00 +CPU= 19.006
  23:40:42 STEP 2728 H=2728:00 +CPU= 12.487
  23:40:54 STEP 2729 H=2729:00 +CPU= 12.330
  23:41:15 STEP 2730 H=2730:00 +CPU= 21.149
  23:41:28 STEP 2731 H=2731:00 +CPU= 12.609
  23:41:42 STEP 2732 H=2732:00 +CPU= 14.005
  23:42:01 STEP 2733 H=2733:00 +CPU= 19.547
  23:42:14 STEP 2734 H=2734:00 +CPU= 13.151
  23:42:27 STEP 2735 H=2735:00 +CPU= 12.812
  23:42:51 STEP 2736 H=2736:00 +CPU= 23.287
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_113.zip
Uploading the intermediate file: upload_file_113.zip
  23:43:03 STEP 2737 H=2737:00 +CPU= 12.101
  23:43:15 STEP 2738 H=2738:00 +CPU= 12.464
  23:43:34 STEP 2739 H=2739:00 +CPU= 19.102
  23:43:47 STEP 2740 H=2740:00 +CPU= 12.649
  23:43:59 STEP 2741 H=2741:00 +CPU= 12.549
  23:44:20 STEP 2742 H=2742:00 +CPU= 20.604
  23:44:32 STEP 2743 H=2743:00 +CPU= 11.803
  23:44:44 STEP 2744 H=2744:00 +CPU= 12.468
  23:45:04 STEP 2745 H=2745:00 +CPU= 19.732
  23:45:16 STEP 2746 H=2746:00 +CPU= 12.220
  23:45:29 STEP 2747 H=2747:00 +CPU= 12.162
  23:45:50 STEP 2748 H=2748:00 +CPU= 21.885
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002748
  23:46:03 STEP 2749 H=2749:00 +CPU= 12.101
  23:46:15 STEP 2750 H=2750:00 +CPU= 12.161
  23:46:34 STEP 2751 H=2751:00 +CPU= 19.362
  23:46:48 STEP 2752 H=2752:00 +CPU= 13.502
  23:46:59 STEP 2753 H=2753:00 +CPU= 11.625
  23:47:20 STEP 2754 H=2754:00 +CPU= 20.897
  23:47:32 STEP 2755 H=2755:00 +CPU= 12.081
  23:47:45 STEP 2756 H=2756:00 +CPU= 12.262
  23:48:04 STEP 2757 H=2757:00 +CPU= 19.453
  23:48:17 STEP 2758 H=2758:00 +CPU= 12.999
  23:48:30 STEP 2759 H=2759:00 +CPU= 12.816
  23:48:55 STEP 2760 H=2760:00 +CPU= 24.973
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_114.zip
Uploading the intermediate file: upload_file_114.zip
  23:49:07 STEP 2761 H=2761:00 +CPU= 11.978
  23:49:21 STEP 2762 H=2762:00 +CPU= 13.786
  23:49:39 STEP 2763 H=2763:00 +CPU= 18.832
  23:49:53 STEP 2764 H=2764:00 +CPU= 13.646
  23:50:05 STEP 2765 H=2765:00 +CPU= 11.408
  23:50:25 STEP 2766 H=2766:00 +CPU= 20.291
  23:50:37 STEP 2767 H=2767:00 +CPU= 12.603
  23:50:51 STEP 2768 H=2768:00 +CPU= 13.555
  23:51:11 STEP 2769 H=2769:00 +CPU= 20.182
  23:51:24 STEP 2770 H=2770:00 +CPU= 12.900
  23:51:36 STEP 2771 H=2771:00 +CPU= 11.744
  23:51:59 STEP 2772 H=2772:00 +CPU= 22.606
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002772
  23:52:10 STEP 2773 H=2773:00 +CPU= 11.738
  23:52:23 STEP 2774 H=2774:00 +CPU= 12.492
  23:52:43 STEP 2775 H=2775:00 +CPU= 20.318
  23:52:56 STEP 2776 H=2776:00 +CPU= 13.288
  23:53:09 STEP 2777 H=2777:00 +CPU= 12.869
  23:53:29 STEP 2778 H=2778:00 +CPU= 19.981
  23:53:41 STEP 2779 H=2779:00 +CPU= 11.840
  23:53:54 STEP 2780 H=2780:00 +CPU= 12.633
  23:54:13 STEP 2781 H=2781:00 +CPU= 19.346
  23:54:25 STEP 2782 H=2782:00 +CPU= 12.352
  23:54:39 STEP 2783 H=2783:00 +CPU= 13.379
  23:55:03 STEP 2784 H=2784:00 +CPU= 23.948
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_115.zip
Uploading the intermediate file: upload_file_115.zip
  23:55:13 STEP 2785 H=2785:00 +CPU= 10.026
  23:55:23 STEP 2786 H=2786:00 +CPU= 10.247
  23:55:39 STEP 2787 H=2787:00 +CPU= 16.042
  23:55:51 STEP 2788 H=2788:00 +CPU= 12.170
  23:56:05 STEP 2789 H=2789:00 +CPU= 13.339
  23:56:25 STEP 2790 H=2790:00 +CPU= 20.210
  23:56:37 STEP 2791 H=2791:00 +CPU= 12.337
  23:56:50 STEP 2792 H=2792:00 +CPU= 12.845
  23:57:10 STEP 2793 H=2793:00 +CPU= 19.996
  23:57:22 STEP 2794 H=2794:00 +CPU= 12.260
  23:57:34 STEP 2795 H=2795:00 +CPU= 12.105
  23:57:57 STEP 2796 H=2796:00 +CPU= 22.791
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002796
  23:58:10 STEP 2797 H=2797:00 +CPU= 12.769
  23:58:22 STEP 2798 H=2798:00 +CPU= 12.335
  23:58:43 STEP 2799 H=2799:00 +CPU= 20.867
  23:58:56 STEP 2800 H=2800:00 +CPU= 13.066
  23:59:08 STEP 2801 H=2801:00 +CPU= 12.078
  23:59:28 STEP 2802 H=2802:00 +CPU= 19.605
  23:59:40 STEP 2803 H=2803:00 +CPU= 11.843
  23:59:53 STEP 2804 H=2804:00 +CPU= 12.676
  00:00:12 STEP 2805 H=2805:00 +CPU= 19.396
  00:00:24 STEP 2806 H=2806:00 +CPU= 12.103
  00:00:37 STEP 2807 H=2807:00 +CPU= 12.730
  00:00:59 STEP 2808 H=2808:00 +CPU= 22.539
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_116.zip
Uploading the intermediate file: upload_file_116.zip
  00:01:12 STEP 2809 H=2809:00 +CPU= 12.861
  00:01:25 STEP 2810 H=2810:00 +CPU= 12.465
  00:01:44 STEP 2811 H=2811:00 +CPU= 19.130
  00:01:56 STEP 2812 H=2812:00 +CPU= 12.472
  00:02:09 STEP 2813 H=2813:00 +CPU= 12.259
  00:02:29 STEP 2814 H=2814:00 +CPU= 19.774
  00:02:41 STEP 2815 H=2815:00 +CPU= 12.408
  00:02:53 STEP 2816 H=2816:00 +CPU= 12.364
  00:03:13 STEP 2817 H=2817:00 +CPU= 19.907
  00:03:26 STEP 2818 H=2818:00 +CPU= 12.459
  00:03:38 STEP 2819 H=2819:00 +CPU= 12.067
  00:04:00 STEP 2820 H=2820:00 +CPU= 21.796
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002820
  00:04:12 STEP 2821 H=2821:00 +CPU= 12.261
  00:04:24 STEP 2822 H=2822:00 +CPU= 12.267
  00:04:44 STEP 2823 H=2823:00 +CPU= 20.293
  00:04:57 STEP 2824 H=2824:00 +CPU= 12.417
  00:05:09 STEP 2825 H=2825:00 +CPU= 12.037
  00:05:29 STEP 2826 H=2826:00 +CPU= 20.083
  00:05:41 STEP 2827 H=2827:00 +CPU= 11.933
  00:05:53 STEP 2828 H=2828:00 +CPU= 12.089
  00:06:13 STEP 2829 H=2829:00 +CPU= 19.618
  00:06:25 STEP 2830 H=2830:00 +CPU= 12.137
  00:06:37 STEP 2831 H=2831:00 +CPU= 12.437
  00:07:01 STEP 2832 H=2832:00 +CPU= 23.369
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_117.zip
Uploading the intermediate file: upload_file_117.zip
  00:07:13 STEP 2833 H=2833:00 +CPU= 12.457
  00:07:27 STEP 2834 H=2834:00 +CPU= 13.399
  00:07:46 STEP 2835 H=2835:00 +CPU= 19.859
  00:07:59 STEP 2836 H=2836:00 +CPU= 12.380
  00:08:10 STEP 2837 H=2837:00 +CPU= 11.685
  00:08:31 STEP 2838 H=2838:00 +CPU= 20.665
  00:08:43 STEP 2839 H=2839:00 +CPU= 11.963
  00:08:55 STEP 2840 H=2840:00 +CPU= 12.063
  00:09:15 STEP 2841 H=2841:00 +CPU= 19.397
  00:09:27 STEP 2842 H=2842:00 +CPU= 12.288
  00:09:39 STEP 2843 H=2843:00 +CPU= 12.448
  00:10:02 STEP 2844 H=2844:00 +CPU= 22.619
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002844
  00:10:14 STEP 2845 H=2845:00 +CPU= 12.401
  00:10:28 STEP 2846 H=2846:00 +CPU= 13.256
  00:10:48 STEP 2847 H=2847:00 +CPU= 20.575
  00:11:01 STEP 2848 H=2848:00 +CPU= 12.518
  00:11:14 STEP 2849 H=2849:00 +CPU= 12.802
  00:11:30 STEP 2850 H=2850:00 +CPU= 16.202
  00:11:40 STEP 2851 H=2851:00 +CPU=  9.688
  00:11:50 STEP 2852 H=2852:00 +CPU= 10.213
  00:12:10 STEP 2853 H=2853:00 +CPU= 19.842
  00:12:23 STEP 2854 H=2854:00 +CPU= 13.115
  00:12:36 STEP 2855 H=2855:00 +CPU= 13.097
  00:13:00 STEP 2856 H=2856:00 +CPU= 24.409
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_118.zip
Uploading the intermediate file: upload_file_118.zip
  00:13:13 STEP 2857 H=2857:00 +CPU= 12.229
  00:13:25 STEP 2858 H=2858:00 +CPU= 12.956
  00:13:46 STEP 2859 H=2859:00 +CPU= 20.305
  00:13:59 STEP 2860 H=2860:00 +CPU= 12.845
  00:14:11 STEP 2861 H=2861:00 +CPU= 12.396
  00:14:32 STEP 2862 H=2862:00 +CPU= 21.256
  00:14:45 STEP 2863 H=2863:00 +CPU= 12.520
  00:14:58 STEP 2864 H=2864:00 +CPU= 13.143
  00:15:18 STEP 2865 H=2865:00 +CPU= 20.016
  00:15:31 STEP 2866 H=2866:00 +CPU= 12.999
  00:15:43 STEP 2867 H=2867:00 +CPU= 12.327
  00:16:06 STEP 2868 H=2868:00 +CPU= 22.753
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002868
  00:16:19 STEP 2869 H=2869:00 +CPU= 12.620
  00:16:32 STEP 2870 H=2870:00 +CPU= 13.537
  00:16:53 STEP 2871 H=2871:00 +CPU= 20.404
  00:17:06 STEP 2872 H=2872:00 +CPU= 13.230
  00:17:19 STEP 2873 H=2873:00 +CPU= 12.687
  00:17:40 STEP 2874 H=2874:00 +CPU= 21.366
  00:17:53 STEP 2875 H=2875:00 +CPU= 12.783
  00:18:06 STEP 2876 H=2876:00 +CPU= 13.151
  00:18:27 STEP 2877 H=2877:00 +CPU= 20.669
  00:18:40 STEP 2878 H=2878:00 +CPU= 13.358
  00:18:53 STEP 2879 H=2879:00 +CPU= 12.881
  00:19:17 STEP 2880 H=2880:00 +CPU= 24.089
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_119.zip
Uploading the intermediate file: upload_file_119.zip
  00:19:31 STEP 2881 H=2881:00 +CPU= 13.381
Uploading trickle at timestep: 10368000
  00:19:44 STEP 2882 H=2882:00 +CPU= 13.168
  00:20:04 STEP 2883 H=2883:00 +CPU= 20.453
  00:20:17 STEP 2884 H=2884:00 +CPU= 12.692
  00:20:29 STEP 2885 H=2885:00 +CPU= 12.546
  00:20:50 STEP 2886 H=2886:00 +CPU= 20.829
  00:21:03 STEP 2887 H=2887:00 +CPU= 12.499
  00:21:16 STEP 2888 H=2888:00 +CPU= 13.310
  00:21:37 STEP 2889 H=2889:00 +CPU= 20.397
  00:21:50 STEP 2890 H=2890:00 +CPU= 13.204
  00:22:03 STEP 2891 H=2891:00 +CPU= 13.078
  00:22:25 STEP 2892 H=2892:00 +CPU= 22.354
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002892
  00:22:38 STEP 2893 H=2893:00 +CPU= 12.701
  00:22:51 STEP 2894 H=2894:00 +CPU= 12.718
  00:23:12 STEP 2895 H=2895:00 +CPU= 20.889
  00:23:25 STEP 2896 H=2896:00 +CPU= 13.332
  00:23:38 STEP 2897 H=2897:00 +CPU= 12.683
  00:23:58 STEP 2898 H=2898:00 +CPU= 20.658
  00:24:11 STEP 2899 H=2899:00 +CPU= 12.517
  00:24:24 STEP 2900 H=2900:00 +CPU= 12.776
  00:24:44 STEP 2901 H=2901:00 +CPU= 20.438
  00:24:57 STEP 2902 H=2902:00 +CPU= 12.607
  00:25:10 STEP 2903 H=2903:00 +CPU= 13.647
  00:25:34 STEP 2904 H=2904:00 +CPU= 23.898
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_120.zip
Uploading the intermediate file: upload_file_120.zip
  00:25:47 STEP 2905 H=2905:00 +CPU= 12.850
  00:26:00 STEP 2906 H=2906:00 +CPU= 13.070
  00:26:21 STEP 2907 H=2907:00 +CPU= 20.422
  00:26:33 STEP 2908 H=2908:00 +CPU= 12.885
  00:26:46 STEP 2909 H=2909:00 +CPU= 12.764
  00:27:07 STEP 2910 H=2910:00 +CPU= 20.536
  00:27:20 STEP 2911 H=2911:00 +CPU= 12.809
  00:27:32 STEP 2912 H=2912:00 +CPU= 12.778
  00:27:53 STEP 2913 H=2913:00 +CPU= 20.304
  00:28:06 STEP 2914 H=2914:00 +CPU= 13.360
  00:28:19 STEP 2915 H=2915:00 +CPU= 12.694
  00:28:42 STEP 2916 H=2916:00 +CPU= 23.086
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002916
  00:28:55 STEP 2917 H=2917:00 +CPU= 12.949
  00:29:08 STEP 2918 H=2918:00 +CPU= 13.196
  00:29:29 STEP 2919 H=2919:00 +CPU= 20.791
  00:29:42 STEP 2920 H=2920:00 +CPU= 12.793
  00:29:55 STEP 2921 H=2921:00 +CPU= 12.885
  00:30:15 STEP 2922 H=2922:00 +CPU= 20.470
  00:30:28 STEP 2923 H=2923:00 +CPU= 12.725
  00:30:41 STEP 2924 H=2924:00 +CPU= 13.488
  00:31:01 STEP 2925 H=2925:00 +CPU= 19.869
  00:31:15 STEP 2926 H=2926:00 +CPU= 13.486
  00:31:28 STEP 2927 H=2927:00 +CPU= 12.911
  00:31:52 STEP 2928 H=2928:00 +CPU= 24.096
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_121.zip
Uploading the intermediate file: upload_file_121.zip
  00:32:04 STEP 2929 H=2929:00 +CPU= 12.394
  00:32:17 STEP 2930 H=2930:00 +CPU= 13.358
  00:32:37 STEP 2931 H=2931:00 +CPU= 19.805
  00:32:51 STEP 2932 H=2932:00 +CPU= 13.297
  00:33:03 STEP 2933 H=2933:00 +CPU= 12.483
  00:33:24 STEP 2934 H=2934:00 +CPU= 20.511
  00:33:36 STEP 2935 H=2935:00 +CPU= 12.753
  00:33:50 STEP 2936 H=2936:00 +CPU= 13.280
  00:34:11 STEP 2937 H=2937:00 +CPU= 20.982
  00:34:24 STEP 2938 H=2938:00 +CPU= 13.047
  00:34:36 STEP 2939 H=2939:00 +CPU= 12.478
  00:35:00 STEP 2940 H=2940:00 +CPU= 23.374
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002940
  00:35:12 STEP 2941 H=2941:00 +CPU= 12.508
  00:35:25 STEP 2942 H=2942:00 +CPU= 12.928
  00:35:45 STEP 2943 H=2943:00 +CPU= 20.248
  00:35:58 STEP 2944 H=2944:00 +CPU= 12.957
  00:36:11 STEP 2945 H=2945:00 +CPU= 12.470
  00:36:32 STEP 2946 H=2946:00 +CPU= 20.802
  00:36:44 STEP 2947 H=2947:00 +CPU= 12.746
  00:36:57 STEP 2948 H=2948:00 +CPU= 12.804
  00:37:17 STEP 2949 H=2949:00 +CPU= 20.198
  00:37:30 STEP 2950 H=2950:00 +CPU= 13.104
  00:37:43 STEP 2951 H=2951:00 +CPU= 12.978
  00:38:07 STEP 2952 H=2952:00 +CPU= 23.501
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12176635/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0991_1990050100_123_959_12176635_0_r379770239_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
00:41:30 (1103): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:23:34 1537625 22261394 oifs_43r3_ps_0991_1990050100_123_959_12176635_0 864,000 2,703 0.0031


©2024 climateprediction.net