climateprediction.net home page
Task 22286608

Task 22286608

Name oifs_43r3_ps_0741_2014050100_123_983_12200385_0
Workunit 12200385
Created 24 Dec 2022, 10:57:47 UTC
Sent 18 Jan 2023, 15:21:00 UTC
Report deadline 19 Mar 2023, 15:21:00 UTC
Received 29 Jan 2023, 6:18:11 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1536052
Run time 19 hours 15 min 44 sec
CPU time 19 hours 15 min 44 sec
Validate state Invalid
Credit 5,080.32
Device peak FLOPS 3.62 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,410.36 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,745.07 MB
Stderr
<core_client_version>7.22.0</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
2/ICMGGhq0f+000540
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000540
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000540
  18:40:38 STEP  541 H= 541:00 +CPU= 47.626
  18:41:28 STEP  542 H= 542:00 +CPU= 48.992
  18:42:44 STEP  543 H= 543:00 +CPU= 75.475
  18:43:34 STEP  544 H= 544:00 +CPU= 49.234
  18:44:19 STEP  545 H= 545:00 +CPU= 45.445
  18:45:37 STEP  546 H= 546:00 +CPU= 76.661
  18:46:25 STEP  547 H= 547:00 +CPU= 47.440
  18:47:15 STEP  548 H= 548:00 +CPU= 50.359
  18:48:30 STEP  549 H= 549:00 +CPU= 74.086
  18:49:20 STEP  550 H= 550:00 +CPU= 49.505
  18:50:09 STEP  551 H= 551:00 +CPU= 48.291
  18:52:51 STEP  552 H= 552:00 +CPU= 84.395
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000552
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000552
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000552
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000528
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000528
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000528
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000540
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000540
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000540
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_22.zip
Uploading the intermediate file: upload_file_22.zip
  18:53:39 STEP  553 H= 553:00 +CPU= 47.530
  18:54:29 STEP  554 H= 554:00 +CPU= 49.481
  18:55:42 STEP  555 H= 555:00 +CPU= 73.124
  18:56:33 STEP  556 H= 556:00 +CPU= 50.249
  18:57:22 STEP  557 H= 557:00 +CPU= 48.796
  18:58:37 STEP  558 H= 558:00 +CPU= 74.811
  18:59:26 STEP  559 H= 559:00 +CPU= 48.748
  19:00:14 STEP  560 H= 560:00 +CPU= 48.375
  19:01:28 STEP  561 H= 561:00 +CPU= 73.876
  19:02:18 STEP  562 H= 562:00 +CPU= 49.751
  19:03:07 STEP  563 H= 563:00 +CPU= 48.230
  19:04:30 STEP  564 H= 564:00 +CPU= 83.146
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000564
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000564
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000564
  19:05:21 STEP  565 H= 565:00 +CPU= 50.100
  19:06:12 STEP  566 H= 566:00 +CPU= 51.156
  19:07:30 STEP  567 H= 567:00 +CPU= 77.554
  19:08:23 STEP  568 H= 568:00 +CPU= 52.593
  19:09:15 STEP  569 H= 569:00 +CPU= 51.576
  19:10:37 STEP  570 H= 570:00 +CPU= 80.682
  19:11:26 STEP  571 H= 571:00 +CPU= 49.503
  19:12:19 STEP  572 H= 572:00 +CPU= 52.295
  19:13:37 STEP  573 H= 573:00 +CPU= 77.391
  19:14:29 STEP  574 H= 574:00 +CPU= 52.037
  19:15:22 STEP  575 H= 575:00 +CPU= 52.323
  19:16:53 STEP  576 H= 576:00 +CPU= 90.171
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000576
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000576
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000576
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000552
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000552
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000552
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000564
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000564
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000564
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_23.zip
Uploading the intermediate file: upload_file_23.zip
  19:17:45 STEP  577 H= 577:00 +CPU= 50.385
  19:18:37 STEP  578 H= 578:00 +CPU= 51.884
  19:19:56 STEP  579 H= 579:00 +CPU= 78.581
  19:20:48 STEP  580 H= 580:00 +CPU= 51.922
  19:21:39 STEP  581 H= 581:00 +CPU= 50.013
  19:23:00 STEP  582 H= 582:00 +CPU= 80.734
  19:23:50 STEP  583 H= 583:00 +CPU= 50.175
  19:24:42 STEP  584 H= 584:00 +CPU= 51.675
  19:26:01 STEP  585 H= 585:00 +CPU= 77.642
  19:26:53 STEP  586 H= 586:00 +CPU= 52.324
  19:27:45 STEP  587 H= 587:00 +CPU= 51.847
  19:29:14 STEP  588 H= 588:00 +CPU= 88.110
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000588
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000588
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000588
  19:30:06 STEP  589 H= 589:00 +CPU= 51.322
  19:30:59 STEP  590 H= 590:00 +CPU= 52.397
  19:32:16 STEP  591 H= 591:00 +CPU= 77.330
  19:33:11 STEP  592 H= 592:00 +CPU= 53.834
  19:34:02 STEP  593 H= 593:00 +CPU= 51.091
  19:35:20 STEP  594 H= 594:00 +CPU= 77.030
  19:36:11 STEP  595 H= 595:00 +CPU= 50.890
  19:37:03 STEP  596 H= 596:00 +CPU= 51.598
  19:38:20 STEP  597 H= 597:00 +CPU= 75.683
  19:39:13 STEP  598 H= 598:00 +CPU= 51.945
  19:40:06 STEP  599 H= 599:00 +CPU= 52.528
  19:41:36 STEP  600 H= 600:00 +CPU= 89.399
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000600
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000600
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000600
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000576
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000576
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000576
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000588
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000588
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000588
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_24.zip
Uploading the intermediate file: upload_file_24.zip
  19:42:28 STEP  601 H= 601:00 +CPU= 51.831
  19:43:19 STEP  602 H= 602:00 +CPU= 51.442
  19:44:38 STEP  603 H= 603:00 +CPU= 77.714
  19:45:30 STEP  604 H= 604:00 +CPU= 51.949
  19:46:19 STEP  605 H= 605:00 +CPU= 48.215
  19:47:38 STEP  606 H= 606:00 +CPU= 78.704
  19:48:29 STEP  607 H= 607:00 +CPU= 50.289
  19:49:20 STEP  608 H= 608:00 +CPU= 51.406
  19:50:37 STEP  609 H= 609:00 +CPU= 76.318
  19:51:30 STEP  610 H= 610:00 +CPU= 52.456
  19:52:20 STEP  611 H= 611:00 +CPU= 49.511
  19:53:48 STEP  612 H= 612:00 +CPU= 87.749
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000612
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000612
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000612
  19:54:39 STEP  613 H= 613:00 +CPU= 50.214
  19:55:29 STEP  614 H= 614:00 +CPU= 50.230
  19:56:48 STEP  615 H= 615:00 +CPU= 77.910
  19:57:40 STEP  616 H= 616:00 +CPU= 51.755
  19:58:30 STEP  617 H= 617:00 +CPU= 50.177
  19:59:51 STEP  618 H= 618:00 +CPU= 80.342
  20:00:42 STEP  619 H= 619:00 +CPU= 50.359
  20:01:35 STEP  620 H= 620:00 +CPU= 52.737
  20:02:52 STEP  621 H= 621:00 +CPU= 76.718
  20:03:43 STEP  622 H= 622:00 +CPU= 50.939
  20:04:36 STEP  623 H= 623:00 +CPU= 51.824
  20:06:06 STEP  624 H= 624:00 +CPU= 90.337
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000624
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000624
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000624
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000600
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000600
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000600
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000612
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000612
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000612
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_25.zip
Uploading the intermediate file: upload_file_25.zip
  20:06:58 STEP  625 H= 625:00 +CPU= 50.235
  20:07:49 STEP  626 H= 626:00 +CPU= 51.098
  20:09:08 STEP  627 H= 627:00 +CPU= 78.364
  20:10:00 STEP  628 H= 628:00 +CPU= 51.661
  20:10:50 STEP  629 H= 629:00 +CPU= 49.667
  20:12:10 STEP  630 H= 630:00 +CPU= 79.510
  20:13:01 STEP  631 H= 631:00 +CPU= 50.654
  20:13:54 STEP  632 H= 632:00 +CPU= 51.754
  20:15:11 STEP  633 H= 633:00 +CPU= 76.492
  20:16:01 STEP  634 H= 634:00 +CPU= 50.329
  20:16:52 STEP  635 H= 635:00 +CPU= 49.927
  20:18:18 STEP  636 H= 636:00 +CPU= 85.802
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000636
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000636
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000636
  20:19:09 STEP  637 H= 637:00 +CPU= 50.419
  20:20:01 STEP  638 H= 638:00 +CPU= 51.220
  20:21:19 STEP  639 H= 639:00 +CPU= 77.625
  20:22:11 STEP  640 H= 640:00 +CPU= 52.242
  20:23:01 STEP  641 H= 641:00 +CPU= 49.441
  20:24:22 STEP  642 H= 642:00 +CPU= 80.591
  20:25:13 STEP  643 H= 643:00 +CPU= 50.666
  20:26:05 STEP  644 H= 644:00 +CPU= 51.689
  20:27:23 STEP  645 H= 645:00 +CPU= 76.763
  20:28:14 STEP  646 H= 646:00 +CPU= 50.804
  20:29:06 STEP  647 H= 647:00 +CPU= 51.682
  20:30:38 STEP  648 H= 648:00 +CPU= 91.924
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000648
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000648
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000648
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000624
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000624
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000624
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000636
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000636
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000636
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_26.zip
Uploading the intermediate file: upload_file_26.zip
  20:31:28 STEP  649 H= 649:00 +CPU= 49.032
  20:32:21 STEP  650 H= 650:00 +CPU= 52.080
  20:33:39 STEP  651 H= 651:00 +CPU= 77.557
  20:34:29 STEP  652 H= 652:00 +CPU= 49.929
  20:35:19 STEP  653 H= 653:00 +CPU= 49.948
  20:36:41 STEP  654 H= 654:00 +CPU= 81.420
  20:37:33 STEP  655 H= 655:00 +CPU= 50.825
  20:38:25 STEP  656 H= 656:00 +CPU= 52.219
  20:39:42 STEP  657 H= 657:00 +CPU= 76.309
  20:40:34 STEP  658 H= 658:00 +CPU= 51.629
  20:41:26 STEP  659 H= 659:00 +CPU= 51.459
  20:42:52 STEP  660 H= 660:00 +CPU= 85.737
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000660
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000660
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000660
  20:43:43 STEP  661 H= 661:00 +CPU= 50.087
  20:44:35 STEP  662 H= 662:00 +CPU= 51.326
  20:45:54 STEP  663 H= 663:00 +CPU= 78.507
  20:46:45 STEP  664 H= 664:00 +CPU= 51.344
  20:47:35 STEP  665 H= 665:00 +CPU= 49.913
  20:48:56 STEP  666 H= 666:00 +CPU= 80.498
  20:49:47 STEP  667 H= 667:00 +CPU= 50.085
  20:50:39 STEP  668 H= 668:00 +CPU= 52.196
  20:51:58 STEP  669 H= 669:00 +CPU= 77.947
  20:52:50 STEP  670 H= 670:00 +CPU= 51.418
  20:53:42 STEP  671 H= 671:00 +CPU= 52.176
  20:55:14 STEP  672 H= 672:00 +CPU= 91.361
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000672
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000672
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000672
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000648
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000648
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000648
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000660
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000660
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000660
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_27.zip
Uploading the intermediate file: upload_file_27.zip
  20:56:06 STEP  673 H= 673:00 +CPU= 51.364
  20:56:59 STEP  674 H= 674:00 +CPU= 52.028
  20:58:16 STEP  675 H= 675:00 +CPU= 76.578
  20:59:07 STEP  676 H= 676:00 +CPU= 50.898
  20:59:58 STEP  677 H= 677:00 +CPU= 50.627
  21:01:20 STEP  678 H= 678:00 +CPU= 81.246
  21:02:10 STEP  679 H= 679:00 +CPU= 50.161
  21:03:03 STEP  680 H= 680:00 +CPU= 52.271
  21:04:21 STEP  681 H= 681:00 +CPU= 77.770
  21:05:12 STEP  682 H= 682:00 +CPU= 50.342
  21:06:07 STEP  683 H= 683:00 +CPU= 50.962
  21:07:47 STEP  684 H= 684:00 +CPU= 92.185
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000684
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000684
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000684
  21:08:44 STEP  685 H= 685:00 +CPU= 51.305
  21:09:36 STEP  686 H= 686:00 +CPU= 51.673
  21:10:56 STEP  687 H= 687:00 +CPU= 78.028
  21:11:46 STEP  688 H= 688:00 +CPU= 49.885
  21:12:38 STEP  689 H= 689:00 +CPU= 50.765
  21:13:59 STEP  690 H= 690:00 +CPU= 80.371
  21:14:50 STEP  691 H= 691:00 +CPU= 49.258
  21:15:42 STEP  692 H= 692:00 +CPU= 51.134
  21:16:57 STEP  693 H= 693:00 +CPU= 71.330
  21:17:40 STEP  694 H= 694:00 +CPU= 42.730
  21:18:26 STEP  695 H= 695:00 +CPU= 45.109
  21:19:45 STEP  696 H= 696:00 +CPU= 77.534
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000696
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000696
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000696
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000672
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000672
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000672
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000684
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000684
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000684
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_28.zip
Uploading the intermediate file: upload_file_28.zip
  21:20:29 STEP  697 H= 697:00 +CPU= 42.674
  21:21:20 STEP  698 H= 698:00 +CPU= 50.712
  21:22:38 STEP  699 H= 699:00 +CPU= 76.179
  21:23:30 STEP  700 H= 700:00 +CPU= 51.022
  21:24:21 STEP  701 H= 701:00 +CPU= 49.975
  21:25:39 STEP  702 H= 702:00 +CPU= 77.419
  21:26:31 STEP  703 H= 703:00 +CPU= 51.021
  21:27:25 STEP  704 H= 704:00 +CPU= 53.337
  21:28:44 STEP  705 H= 705:00 +CPU= 77.952
  21:29:37 STEP  706 H= 706:00 +CPU= 51.238
  21:30:29 STEP  707 H= 707:00 +CPU= 51.043
  21:31:55 STEP  708 H= 708:00 +CPU= 84.791
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000708
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000708
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000708
  21:32:45 STEP  709 H= 709:00 +CPU= 48.873
  21:33:38 STEP  710 H= 710:00 +CPU= 51.940
  21:34:56 STEP  711 H= 711:00 +CPU= 76.972
  21:35:49 STEP  712 H= 712:00 +CPU= 52.005
  21:36:39 STEP  713 H= 713:00 +CPU= 49.233
  21:37:59 STEP  714 H= 714:00 +CPU= 78.058
  21:38:50 STEP  715 H= 715:00 +CPU= 51.218
  21:39:43 STEP  716 H= 716:00 +CPU= 52.578
  21:41:02 STEP  717 H= 717:00 +CPU= 78.016
  21:41:55 STEP  718 H= 718:00 +CPU= 51.762
  21:42:49 STEP  719 H= 719:00 +CPU= 52.984
  21:44:28 STEP  720 H= 720:00 +CPU= 91.251
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000720
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000720
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000720
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000696
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000696
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000696
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000708
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000708
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000708
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_29.zip
Uploading the intermediate file: upload_file_29.zip
  21:45:18 STEP  721 H= 721:00 +CPU= 50.266
  21:46:11 STEP  722 H= 722:00 +CPU= 52.255
  21:47:27 STEP  723 H= 723:00 +CPU= 76.048
  21:48:20 STEP  724 H= 724:00 +CPU= 52.164
  21:49:11 STEP  725 H= 725:00 +CPU= 50.782
  21:50:29 STEP  726 H= 726:00 +CPU= 78.068
  21:51:20 STEP  727 H= 727:00 +CPU= 50.086
  21:52:13 STEP  728 H= 728:00 +CPU= 52.900
  21:53:30 STEP  729 H= 729:00 +CPU= 76.723
  21:54:23 STEP  730 H= 730:00 +CPU= 52.152
  21:55:14 STEP  731 H= 731:00 +CPU= 50.670
  21:56:41 STEP  732 H= 732:00 +CPU= 86.776
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000732
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000732
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000732
  21:57:32 STEP  733 H= 733:00 +CPU= 50.015
  21:58:24 STEP  734 H= 734:00 +CPU= 51.989
  21:59:43 STEP  735 H= 735:00 +CPU= 77.713
  22:00:34 STEP  736 H= 736:00 +CPU= 51.173
  22:01:25 STEP  737 H= 737:00 +CPU= 50.422
  22:02:45 STEP  738 H= 738:00 +CPU= 79.480
  22:03:36 STEP  739 H= 739:00 +CPU= 50.419
  22:04:27 STEP  740 H= 740:00 +CPU= 51.118
  22:05:44 STEP  741 H= 741:00 +CPU= 76.119
  22:06:36 STEP  742 H= 742:00 +CPU= 51.928
  22:07:28 STEP  743 H= 743:00 +CPU= 51.518
  22:09:01 STEP  744 H= 744:00 +CPU= 91.377
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000744
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000744
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000744
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000720
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000720
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000720
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000732
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000732
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000732
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_30.zip
Uploading the intermediate file: upload_file_30.zip
  22:09:51 STEP  745 H= 745:00 +CPU= 49.943
  22:10:43 STEP  746 H= 746:00 +CPU= 52.205
  22:12:02 STEP  747 H= 747:00 +CPU= 77.697
  22:12:53 STEP  748 H= 748:00 +CPU= 51.078
  22:13:44 STEP  749 H= 749:00 +CPU= 50.686
  22:15:04 STEP  750 H= 750:00 +CPU= 79.207
  22:15:54 STEP  751 H= 751:00 +CPU= 49.261
  22:16:46 STEP  752 H= 752:00 +CPU= 52.007
  22:18:03 STEP  753 H= 753:00 +CPU= 76.402
  22:18:54 STEP  754 H= 754:00 +CPU= 50.517
  22:19:45 STEP  755 H= 755:00 +CPU= 51.092
  22:21:12 STEP  756 H= 756:00 +CPU= 85.715
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000756
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000756
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000756
  22:22:03 STEP  757 H= 757:00 +CPU= 51.525
  22:22:56 STEP  758 H= 758:00 +CPU= 52.646
  22:24:14 STEP  759 H= 759:00 +CPU= 76.924
  22:25:07 STEP  760 H= 760:00 +CPU= 52.528
  22:25:58 STEP  761 H= 761:00 +CPU= 51.286
  22:27:19 STEP  762 H= 762:00 +CPU= 80.064
  22:28:09 STEP  763 H= 763:00 +CPU= 49.332
  22:29:01 STEP  764 H= 764:00 +CPU= 52.071
  22:30:17 STEP  765 H= 765:00 +CPU= 75.604
  22:31:09 STEP  766 H= 766:00 +CPU= 51.189
  22:32:01 STEP  767 H= 767:00 +CPU= 52.244
  22:33:33 STEP  768 H= 768:00 +CPU= 90.269
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000768
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000768
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000768
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000744
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000744
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000744
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000756
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000756
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000756
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_31.zip
Uploading the intermediate file: upload_file_31.zip
Uploading trickle at timestep: 2764800
  22:34:24 STEP  769 H= 769:00 +CPU= 49.821
  22:35:16 STEP  770 H= 770:00 +CPU= 51.914
  22:36:34 STEP  771 H= 771:00 +CPU= 77.082
  22:37:27 STEP  772 H= 772:00 +CPU= 52.479
  22:38:18 STEP  773 H= 773:00 +CPU= 50.636
  22:39:36 STEP  774 H= 774:00 +CPU= 77.975
  22:40:28 STEP  775 H= 775:00 +CPU= 51.111
  22:41:20 STEP  776 H= 776:00 +CPU= 52.200
  22:42:38 STEP  777 H= 777:00 +CPU= 77.560
  22:43:32 STEP  778 H= 778:00 +CPU= 52.761
  22:44:23 STEP  779 H= 779:00 +CPU= 51.240
  22:45:50 STEP  780 H= 780:00 +CPU= 85.872
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000780
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000780
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000780
  22:46:41 STEP  781 H= 781:00 +CPU= 51.129
  22:47:34 STEP  782 H= 782:00 +CPU= 51.982
  22:48:50 STEP  783 H= 783:00 +CPU= 76.083
  22:49:43 STEP  784 H= 784:00 +CPU= 52.657
  22:50:35 STEP  785 H= 785:00 +CPU= 51.616
  22:51:54 STEP  786 H= 786:00 +CPU= 78.186
  22:52:45 STEP  787 H= 787:00 +CPU= 50.533
  22:53:37 STEP  788 H= 788:00 +CPU= 52.079
  22:54:55 STEP  789 H= 789:00 +CPU= 77.288
  22:55:45 STEP  790 H= 790:00 +CPU= 49.559
  22:56:30 STEP  791 H= 791:00 +CPU= 45.100
  22:57:49 STEP  792 H= 792:00 +CPU= 77.456
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000792
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000792
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000792
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000768
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000768
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000768
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000780
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000780
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000780
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_32.zip
Uploading the intermediate file: upload_file_32.zip
  22:58:32 STEP  793 H= 793:00 +CPU= 42.662
  22:59:17 STEP  794 H= 794:00 +CPU= 44.619
  23:00:33 STEP  795 H= 795:00 +CPU= 75.851
  23:01:26 STEP  796 H= 796:00 +CPU= 52.512
  23:02:18 STEP  797 H= 797:00 +CPU= 51.563
  23:03:39 STEP  798 H= 798:00 +CPU= 79.820
  23:04:30 STEP  799 H= 799:00 +CPU= 50.657
  23:05:22 STEP  800 H= 800:00 +CPU= 52.216
  23:06:40 STEP  801 H= 801:00 +CPU= 76.909
  23:07:31 STEP  802 H= 802:00 +CPU= 51.407
  23:08:23 STEP  803 H= 803:00 +CPU= 51.575
  23:09:50 STEP  804 H= 804:00 +CPU= 85.858
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000804
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000804
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000804
  23:10:41 STEP  805 H= 805:00 +CPU= 51.046
  23:11:34 STEP  806 H= 806:00 +CPU= 51.994
  23:12:50 STEP  807 H= 807:00 +CPU= 76.165
  23:13:43 STEP  808 H= 808:00 +CPU= 52.342
  23:14:35 STEP  809 H= 809:00 +CPU= 51.261
  23:15:53 STEP  810 H= 810:00 +CPU= 78.071
  23:16:44 STEP  811 H= 811:00 +CPU= 50.253
  23:17:36 STEP  812 H= 812:00 +CPU= 51.232
  23:18:53 STEP  813 H= 813:00 +CPU= 76.592
  23:19:45 STEP  814 H= 814:00 +CPU= 52.031
  23:20:38 STEP  815 H= 815:00 +CPU= 52.497
  23:22:10 STEP  816 H= 816:00 +CPU= 90.185
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000816
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000816
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000816
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000792
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000792
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000792
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000804
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000804
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000804
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_33.zip
Uploading the intermediate file: upload_file_33.zip
  23:23:02 STEP  817 H= 817:00 +CPU= 50.520
  23:23:52 STEP  818 H= 818:00 +CPU= 50.394
  23:25:08 STEP  819 H= 819:00 +CPU= 75.443
  23:26:00 STEP  820 H= 820:00 +CPU= 51.720
  23:26:50 STEP  821 H= 821:00 +CPU= 49.409
  23:28:09 STEP  822 H= 822:00 +CPU= 78.067
  23:29:01 STEP  823 H= 823:00 +CPU= 51.477
  23:29:54 STEP  824 H= 824:00 +CPU= 52.754
  23:31:13 STEP  825 H= 825:00 +CPU= 77.829
  23:32:05 STEP  826 H= 826:00 +CPU= 51.978
  23:32:57 STEP  827 H= 827:00 +CPU= 51.643
  23:34:22 STEP  828 H= 828:00 +CPU= 84.551
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000828
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000828
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000828
  23:35:12 STEP  829 H= 829:00 +CPU= 49.719
  23:36:05 STEP  830 H= 830:00 +CPU= 52.176
  23:37:23 STEP  831 H= 831:00 +CPU= 77.809
  23:38:15 STEP  832 H= 832:00 +CPU= 51.586
  23:39:07 STEP  833 H= 833:00 +CPU= 51.461
  23:40:27 STEP  834 H= 834:00 +CPU= 78.920
  23:41:17 STEP  835 H= 835:00 +CPU= 49.783
  23:42:09 STEP  836 H= 836:00 +CPU= 51.955
  23:43:26 STEP  837 H= 837:00 +CPU= 76.322
  23:44:18 STEP  838 H= 838:00 +CPU= 52.043
  23:45:12 STEP  839 H= 839:00 +CPU= 53.198
  23:46:44 STEP  840 H= 840:00 +CPU= 90.940
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000840
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000840
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000840
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000816
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000816
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000816
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000828
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000828
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000828
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_34.zip
Uploading the intermediate file: upload_file_34.zip
  23:47:35 STEP  841 H= 841:00 +CPU= 50.187
  23:48:26 STEP  842 H= 842:00 +CPU= 50.833
  23:49:44 STEP  843 H= 843:00 +CPU= 77.469
  23:50:36 STEP  844 H= 844:00 +CPU= 51.318
  23:51:27 STEP  845 H= 845:00 +CPU= 50.863
  23:52:46 STEP  846 H= 846:00 +CPU= 78.365
  23:53:37 STEP  847 H= 847:00 +CPU= 50.666
  23:54:30 STEP  848 H= 848:00 +CPU= 52.837
  23:55:47 STEP  849 H= 849:00 +CPU= 76.378
  23:56:40 STEP  850 H= 850:00 +CPU= 51.804
  23:57:31 STEP  851 H= 851:00 +CPU= 51.197
  23:58:58 STEP  852 H= 852:00 +CPU= 85.738
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000852
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000852
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000852
  23:59:48 STEP  853 H= 853:00 +CPU= 50.501
  00:00:41 STEP  854 H= 854:00 +CPU= 51.678
  00:01:59 STEP  855 H= 855:00 +CPU= 77.953
  00:02:52 STEP  856 H= 856:00 +CPU= 52.690
  00:03:43 STEP  857 H= 857:00 +CPU= 50.405
  00:05:04 STEP  858 H= 858:00 +CPU= 80.127
  00:05:53 STEP  859 H= 859:00 +CPU= 49.102
  00:06:46 STEP  860 H= 860:00 +CPU= 52.211
  00:08:04 STEP  861 H= 861:00 +CPU= 77.584
  00:08:57 STEP  862 H= 862:00 +CPU= 52.939
  00:09:50 STEP  863 H= 863:00 +CPU= 52.717
  00:11:22 STEP  864 H= 864:00 +CPU= 90.495
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000864
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000864
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000864
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000840
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000840
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000840
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000852
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000852
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000852
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_35.zip
Uploading the intermediate file: upload_file_35.zip
  00:12:13 STEP  865 H= 865:00 +CPU= 50.126
  00:13:06 STEP  866 H= 866:00 +CPU= 52.320
  00:14:24 STEP  867 H= 867:00 +CPU= 77.263
  00:15:18 STEP  868 H= 868:00 +CPU= 53.020
  00:16:08 STEP  869 H= 869:00 +CPU= 49.866
  00:17:27 STEP  870 H= 870:00 +CPU= 78.310
  00:18:17 STEP  871 H= 871:00 +CPU= 50.014
  00:19:09 STEP  872 H= 872:00 +CPU= 51.665
  00:20:27 STEP  873 H= 873:00 +CPU= 77.379
  00:21:20 STEP  874 H= 874:00 +CPU= 52.141
  00:22:11 STEP  875 H= 875:00 +CPU= 51.004
  00:23:37 STEP  876 H= 876:00 +CPU= 85.514
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000876
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000876
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000876
  00:24:28 STEP  877 H= 877:00 +CPU= 50.353
  00:25:20 STEP  878 H= 878:00 +CPU= 51.615
  00:26:39 STEP  879 H= 879:00 +CPU= 78.282
  00:27:32 STEP  880 H= 880:00 +CPU= 52.108
  00:28:22 STEP  881 H= 881:00 +CPU= 50.176
  00:29:42 STEP  882 H= 882:00 +CPU= 78.822
  00:30:32 STEP  883 H= 883:00 +CPU= 49.911
  00:31:23 STEP  884 H= 884:00 +CPU= 50.921
  00:32:41 STEP  885 H= 885:00 +CPU= 77.219
  00:33:35 STEP  886 H= 886:00 +CPU= 52.844
  00:34:28 STEP  887 H= 887:00 +CPU= 52.010
  00:36:01 STEP  888 H= 888:00 +CPU= 90.208
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000888
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000888
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000888
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000864
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000864
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000864
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000876
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000876
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000876
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_36.zip
Uploading the intermediate file: upload_file_36.zip
  00:36:54 STEP  889 H= 889:00 +CPU= 50.868
  00:37:44 STEP  890 H= 890:00 +CPU= 50.436
  00:39:03 STEP  891 H= 891:00 +CPU= 76.834
  00:39:55 STEP  892 H= 892:00 +CPU= 52.288
  00:40:46 STEP  893 H= 893:00 +CPU= 50.726
  00:42:04 STEP  894 H= 894:00 +CPU= 76.996
  00:42:55 STEP  895 H= 895:00 +CPU= 50.820
  00:43:46 STEP  896 H= 896:00 +CPU= 51.102
  00:45:05 STEP  897 H= 897:00 +CPU= 77.768
  00:45:58 STEP  898 H= 898:00 +CPU= 52.983
  00:46:50 STEP  899 H= 899:00 +CPU= 51.299
  00:48:16 STEP  900 H= 900:00 +CPU= 85.341
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000900
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000900
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000900
  00:49:06 STEP  901 H= 901:00 +CPU= 50.408
  00:49:57 STEP  902 H= 902:00 +CPU= 50.807
  00:51:16 STEP  903 H= 903:00 +CPU= 77.770
  00:52:09 STEP  904 H= 904:00 +CPU= 52.813
  00:53:00 STEP  905 H= 905:00 +CPU= 50.691
  00:54:19 STEP  906 H= 906:00 +CPU= 78.877
  00:55:10 STEP  907 H= 907:00 +CPU= 50.725
  00:56:02 STEP  908 H= 908:00 +CPU= 51.765
  00:57:21 STEP  909 H= 909:00 +CPU= 77.927
  00:58:14 STEP  910 H= 910:00 +CPU= 52.381
  00:59:07 STEP  911 H= 911:00 +CPU= 52.356
  01:00:40 STEP  912 H= 912:00 +CPU= 90.191
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000912
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000912
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000912
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000888
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000888
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000888
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000900
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000900
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000900
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_37.zip
Uploading the intermediate file: upload_file_37.zip
  01:01:31 STEP  913 H= 913:00 +CPU= 50.548
  01:02:23 STEP  914 H= 914:00 +CPU= 51.516
  01:03:42 STEP  915 H= 915:00 +CPU= 77.710
  01:04:34 STEP  916 H= 916:00 +CPU= 52.338
  01:05:25 STEP  917 H= 917:00 +CPU= 50.670
  01:06:45 STEP  918 H= 918:00 +CPU= 79.023
  01:07:36 STEP  919 H= 919:00 +CPU= 50.771
  01:08:27 STEP  920 H= 920:00 +CPU= 50.967
  01:09:45 STEP  921 H= 921:00 +CPU= 77.112
  01:10:38 STEP  922 H= 922:00 +CPU= 52.986
  01:11:29 STEP  923 H= 923:00 +CPU= 50.440
  01:12:56 STEP  924 H= 924:00 +CPU= 86.577
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000924
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000924
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000924
  01:13:47 STEP  925 H= 925:00 +CPU= 50.556
  01:14:38 STEP  926 H= 926:00 +CPU= 50.638
  01:15:57 STEP  927 H= 927:00 +CPU= 77.948
  01:16:50 STEP  928 H= 928:00 +CPU= 52.652
  01:17:40 STEP  929 H= 929:00 +CPU= 49.885
  01:18:59 STEP  930 H= 930:00 +CPU= 78.514
  01:19:50 STEP  931 H= 931:00 +CPU= 50.446
  01:20:42 STEP  932 H= 932:00 +CPU= 51.857
  01:22:01 STEP  933 H= 933:00 +CPU= 77.585
  01:22:53 STEP  934 H= 934:00 +CPU= 51.538
  01:23:46 STEP  935 H= 935:00 +CPU= 50.764
  01:25:18 STEP  936 H= 936:00 +CPU= 89.971
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000936
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000936
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000936
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000912
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000912
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000912
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000924
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000924
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000924
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_38.zip
Uploading the intermediate file: upload_file_38.zip
  01:26:08 STEP  937 H= 937:00 +CPU= 49.703
  01:27:00 STEP  938 H= 938:00 +CPU= 50.955
  01:28:19 STEP  939 H= 939:00 +CPU= 77.653
  01:29:11 STEP  940 H= 940:00 +CPU= 52.454
  01:30:03 STEP  941 H= 941:00 +CPU= 50.699
  01:31:23 STEP  942 H= 942:00 +CPU= 79.986
  01:32:14 STEP  943 H= 943:00 +CPU= 50.609
  01:33:06 STEP  944 H= 944:00 +CPU= 51.826
  01:34:24 STEP  945 H= 945:00 +CPU= 76.613
  01:35:17 STEP  946 H= 946:00 +CPU= 52.588
  01:36:07 STEP  947 H= 947:00 +CPU= 49.737
  01:37:33 STEP  948 H= 948:00 +CPU= 85.253
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000948
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000948
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000948
  01:38:24 STEP  949 H= 949:00 +CPU= 51.153
  01:39:17 STEP  950 H= 950:00 +CPU= 51.845
  01:40:33 STEP  951 H= 951:00 +CPU= 76.277
  01:41:25 STEP  952 H= 952:00 +CPU= 51.478
  01:42:15 STEP  953 H= 953:00 +CPU= 49.860
  01:43:34 STEP  954 H= 954:00 +CPU= 77.912
  01:44:24 STEP  955 H= 955:00 +CPU= 50.327
  01:45:16 STEP  956 H= 956:00 +CPU= 51.346
  01:46:34 STEP  957 H= 957:00 +CPU= 77.234
  01:47:26 STEP  958 H= 958:00 +CPU= 51.689
  01:48:18 STEP  959 H= 959:00 +CPU= 51.197
  01:49:48 STEP  960 H= 960:00 +CPU= 89.453
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000960
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000960
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000960
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000936
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000936
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000936
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000948
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000948
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000948
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_39.zip
Uploading the intermediate file: upload_file_39.zip
  01:50:39 STEP  961 H= 961:00 +CPU= 49.953
  01:51:29 STEP  962 H= 962:00 +CPU= 49.767
  01:52:46 STEP  963 H= 963:00 +CPU= 76.009
  01:53:37 STEP  964 H= 964:00 +CPU= 50.622
  01:54:26 STEP  965 H= 965:00 +CPU= 49.018
  01:55:46 STEP  966 H= 966:00 +CPU= 78.673
  01:56:37 STEP  967 H= 967:00 +CPU= 50.909
  01:57:29 STEP  968 H= 968:00 +CPU= 51.962
  01:58:46 STEP  969 H= 969:00 +CPU= 76.577
  01:59:39 STEP  970 H= 970:00 +CPU= 51.982
  02:00:30 STEP  971 H= 971:00 +CPU= 50.637
  02:01:55 STEP  972 H= 972:00 +CPU= 84.793
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000972
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000972
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000972
  02:02:46 STEP  973 H= 973:00 +CPU= 50.348
  02:03:37 STEP  974 H= 974:00 +CPU= 51.323
  02:04:55 STEP  975 H= 975:00 +CPU= 76.740
  02:05:46 STEP  976 H= 976:00 +CPU= 51.354
  02:06:36 STEP  977 H= 977:00 +CPU= 48.856
  02:07:55 STEP  978 H= 978:00 +CPU= 78.553
  02:08:44 STEP  979 H= 979:00 +CPU= 49.100
  02:09:36 STEP  980 H= 980:00 +CPU= 51.138
  02:10:53 STEP  981 H= 981:00 +CPU= 76.588
  02:11:45 STEP  982 H= 982:00 +CPU= 51.558
  02:12:38 STEP  983 H= 983:00 +CPU= 52.426
  02:14:09 STEP  984 H= 984:00 +CPU= 89.551
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000984
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000984
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000984
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000960
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000960
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000960
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000972
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000972
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000972
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_40.zip
Uploading the intermediate file: upload_file_40.zip
  02:15:00 STEP  985 H= 985:00 +CPU= 50.049
  02:15:51 STEP  986 H= 986:00 +CPU= 50.759
  02:17:09 STEP  987 H= 987:00 +CPU= 77.177
  02:18:02 STEP  988 H= 988:00 +CPU= 52.614
  02:18:52 STEP  989 H= 989:00 +CPU= 49.599
  02:20:11 STEP  990 H= 990:00 +CPU= 78.413
  02:21:01 STEP  991 H= 991:00 +CPU= 50.353
  02:21:54 STEP  992 H= 992:00 +CPU= 52.303
  02:23:13 STEP  993 H= 993:00 +CPU= 77.871
  02:24:05 STEP  994 H= 994:00 +CPU= 51.497
  02:24:55 STEP  995 H= 995:00 +CPU= 49.829
  02:26:20 STEP  996 H= 996:00 +CPU= 84.273
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+000996
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+000996
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+000996
  02:27:11 STEP  997 H= 997:00 +CPU= 51.122
  02:28:02 STEP  998 H= 998:00 +CPU= 50.611
  02:29:20 STEP  999 H= 999:00 +CPU= 76.904
  02:30:11 STEP 1000 H=1000:00 +CPU= 51.252
  02:31:01 STEP 1001 H=1001:00 +CPU= 49.366
  02:32:21 STEP 1002 H=1002:00 +CPU= 76.167
  02:33:13 STEP 1003 H=1003:00 +CPU= 51.733
  02:34:07 STEP 1004 H=1004:00 +CPU= 53.299
  02:35:25 STEP 1005 H=1005:00 +CPU= 76.345
  02:36:17 STEP 1006 H=1006:00 +CPU= 51.297
  02:37:14 STEP 1007 H=1007:00 +CPU= 54.263
  02:38:47 STEP 1008 H=1008:00 +CPU= 87.424
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001008
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001008
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001008
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000984
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000984
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000984
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+000996
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+000996
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+000996
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_41.zip
Uploading the intermediate file: upload_file_41.zip
Uploading trickle at timestep: 3628800
  02:39:39 STEP 1009 H=1009:00 +CPU= 51.390
  02:40:32 STEP 1010 H=1010:00 +CPU= 51.203
  02:42:06 STEP 1011 H=1011:00 +CPU= 72.459
  02:43:01 STEP 1012 H=1012:00 +CPU= 48.532
  02:43:54 STEP 1013 H=1013:00 +CPU= 50.911
  02:45:16 STEP 1014 H=1014:00 +CPU= 78.852
  02:46:09 STEP 1015 H=1015:00 +CPU= 51.707
  02:47:01 STEP 1016 H=1016:00 +CPU= 48.934
  02:48:19 STEP 1017 H=1017:00 +CPU= 77.301
  02:49:12 STEP 1018 H=1018:00 +CPU= 52.439
  02:50:04 STEP 1019 H=1019:00 +CPU= 47.437
  02:52:29 STEP 1020 H=1020:00 +CPU= 89.004
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001020
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001020
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001020
  02:53:54 STEP 1021 H=1021:00 +CPU= 53.691
  02:55:19 STEP 1022 H=1022:00 +CPU= 55.195
  02:56:52 STEP 1023 H=1023:00 +CPU= 80.504
  02:57:54 STEP 1024 H=1024:00 +CPU= 53.627
  02:58:54 STEP 1025 H=1025:00 +CPU= 52.524
  03:00:28 STEP 1026 H=1026:00 +CPU= 81.588
  03:01:29 STEP 1027 H=1027:00 +CPU= 53.134
  03:02:31 STEP 1028 H=1028:00 +CPU= 54.160
  03:04:00 STEP 1029 H=1029:00 +CPU= 78.909
  03:05:01 STEP 1030 H=1030:00 +CPU= 53.775
  03:06:02 STEP 1031 H=1031:00 +CPU= 52.161
  03:08:26 STEP 1032 H=1032:00 +CPU= 91.349
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001032
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001032
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001032
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001008
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001008
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001008
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001020
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001020
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001020
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_42.zip
Uploading the intermediate file: upload_file_42.zip
  03:09:29 STEP 1033 H=1033:00 +CPU= 52.418
  03:10:29 STEP 1034 H=1034:00 +CPU= 48.688
  03:11:48 STEP 1035 H=1035:00 +CPU= 68.942
  03:12:47 STEP 1036 H=1036:00 +CPU= 52.471
  03:13:46 STEP 1037 H=1037:00 +CPU= 51.912
  03:15:21 STEP 1038 H=1038:00 +CPU= 82.915
  03:16:22 STEP 1039 H=1039:00 +CPU= 53.024
  03:17:22 STEP 1040 H=1040:00 +CPU= 53.212
  03:18:55 STEP 1041 H=1041:00 +CPU= 81.122
  03:19:56 STEP 1042 H=1042:00 +CPU= 53.672
  03:20:55 STEP 1043 H=1043:00 +CPU= 51.934
  03:22:38 STEP 1044 H=1044:00 +CPU= 90.784
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001044
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001044
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001044
  03:23:37 STEP 1045 H=1045:00 +CPU= 52.133
  03:24:38 STEP 1046 H=1046:00 +CPU= 53.790
  03:26:12 STEP 1047 H=1047:00 +CPU= 81.305
  03:27:13 STEP 1048 H=1048:00 +CPU= 53.626
  03:28:12 STEP 1049 H=1049:00 +CPU= 52.092
  03:29:47 STEP 1050 H=1050:00 +CPU= 83.357
  03:30:46 STEP 1051 H=1051:00 +CPU= 52.368
  03:31:47 STEP 1052 H=1052:00 +CPU= 53.241
  03:33:23 STEP 1053 H=1053:00 +CPU= 79.357
  03:34:24 STEP 1054 H=1054:00 +CPU= 53.839
  03:36:07 STEP 1055 H=1055:00 +CPU= 52.154
  03:37:55 STEP 1056 H=1056:00 +CPU= 84.753
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001056
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001056
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001056
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001032
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001032
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001032
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001044
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001044
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001044
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_43.zip
Uploading the intermediate file: upload_file_43.zip
  03:38:55 STEP 1057 H=1057:00 +CPU= 52.022
  03:39:56 STEP 1058 H=1058:00 +CPU= 53.249
  03:41:26 STEP 1059 H=1059:00 +CPU= 71.756
  03:43:37 STEP 1060 H=1060:00 +CPU= 50.595
  03:44:35 STEP 1061 H=1061:00 +CPU= 50.361
  03:46:11 STEP 1062 H=1062:00 +CPU= 81.729
  03:47:12 STEP 1063 H=1063:00 +CPU= 53.090
  03:48:14 STEP 1064 H=1064:00 +CPU= 54.333
  03:49:50 STEP 1065 H=1065:00 +CPU= 79.829
  03:50:53 STEP 1066 H=1066:00 +CPU= 54.747
  03:51:53 STEP 1067 H=1067:00 +CPU= 52.442
  03:53:38 STEP 1068 H=1068:00 +CPU= 89.929
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001068
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001068
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001068
  03:54:37 STEP 1069 H=1069:00 +CPU= 51.987
  03:55:38 STEP 1070 H=1070:00 +CPU= 53.904
  03:57:10 STEP 1071 H=1071:00 +CPU= 81.366
  03:58:13 STEP 1072 H=1072:00 +CPU= 55.272
  03:59:11 STEP 1073 H=1073:00 +CPU= 52.362
  05:32:05 STEP 1074 H=1074:00 +CPU= 78.225
  05:33:04 STEP 1075 H=1075:00 +CPU= 49.982
  05:34:00 STEP 1076 H=1076:00 +CPU= 49.652
  05:35:31 STEP 1077 H=1077:00 +CPU= 80.328
  05:36:32 STEP 1078 H=1078:00 +CPU= 54.028
  05:37:36 STEP 1079 H=1079:00 +CPU= 53.056
  05:39:42 STEP 1080 H=1080:00 +CPU= 81.254
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001080
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001080
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001080
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001056
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001056
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001056
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001068
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001068
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001068
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_44.zip
Uploading the intermediate file: upload_file_44.zip
  05:40:41 STEP 1081 H=1081:00 +CPU= 52.548
  05:41:41 STEP 1082 H=1082:00 +CPU= 52.230
  05:43:16 STEP 1083 H=1083:00 +CPU= 80.981
  05:44:18 STEP 1084 H=1084:00 +CPU= 52.411
  05:45:13 STEP 1085 H=1085:00 +CPU= 49.122
  05:46:42 STEP 1086 H=1086:00 +CPU= 80.601
  05:47:43 STEP 1087 H=1087:00 +CPU= 53.266
  05:48:45 STEP 1088 H=1088:00 +CPU= 54.255
  05:50:18 STEP 1089 H=1089:00 +CPU= 80.982
  05:51:19 STEP 1090 H=1090:00 +CPU= 53.945
  05:52:17 STEP 1091 H=1091:00 +CPU= 52.503
  05:53:59 STEP 1092 H=1092:00 +CPU= 89.618
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001092
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001092
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001092
  05:54:58 STEP 1093 H=1093:00 +CPU= 52.198
  05:55:58 STEP 1094 H=1094:00 +CPU= 53.541
  05:57:29 STEP 1095 H=1095:00 +CPU= 80.773
  05:58:31 STEP 1096 H=1096:00 +CPU= 54.536
  05:59:31 STEP 1097 H=1097:00 +CPU= 52.714
  06:01:04 STEP 1098 H=1098:00 +CPU= 81.957
  06:02:00 STEP 1099 H=1099:00 +CPU= 50.620
  06:03:01 STEP 1100 H=1100:00 +CPU= 53.566
  06:04:33 STEP 1101 H=1101:00 +CPU= 80.436
  06:05:36 STEP 1102 H=1102:00 +CPU= 54.742
  06:06:38 STEP 1103 H=1103:00 +CPU= 50.951
  06:08:48 STEP 1104 H=1104:00 +CPU= 93.753
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001104
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001104
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001104
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001080
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001080
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001080
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001092
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001092
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001092
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_45.zip
Uploading the intermediate file: upload_file_45.zip
  06:09:45 STEP 1105 H=1105:00 +CPU= 49.151
  06:10:42 STEP 1106 H=1106:00 +CPU= 50.943
  06:12:15 STEP 1107 H=1107:00 +CPU= 81.780
  06:13:14 STEP 1108 H=1108:00 +CPU= 53.875
  06:14:15 STEP 1109 H=1109:00 +CPU= 50.333
  06:15:48 STEP 1110 H=1110:00 +CPU= 83.219
  06:16:49 STEP 1111 H=1111:00 +CPU= 52.032
  06:17:48 STEP 1112 H=1112:00 +CPU= 52.983
  06:19:23 STEP 1113 H=1113:00 +CPU= 82.597
  06:20:24 STEP 1114 H=1114:00 +CPU= 54.133
  06:21:24 STEP 1115 H=1115:00 +CPU= 52.064
  06:23:06 STEP 1116 H=1116:00 +CPU= 90.543
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001116
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001116
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001116
  06:24:06 STEP 1117 H=1117:00 +CPU= 52.384
  06:25:07 STEP 1118 H=1118:00 +CPU= 53.634
  06:26:39 STEP 1119 H=1119:00 +CPU= 82.030
  06:27:40 STEP 1120 H=1120:00 +CPU= 53.245
  06:28:40 STEP 1121 H=1121:00 +CPU= 51.779
  06:30:11 STEP 1122 H=1122:00 +CPU= 81.241
  06:31:12 STEP 1123 H=1123:00 +CPU= 52.048
  06:32:13 STEP 1124 H=1124:00 +CPU= 53.371
  06:33:46 STEP 1125 H=1125:00 +CPU= 81.539
  06:34:48 STEP 1126 H=1126:00 +CPU= 54.222
  06:35:48 STEP 1127 H=1127:00 +CPU= 53.602
  06:37:40 STEP 1128 H=1128:00 +CPU= 95.558
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001128
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001128
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001128
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001104
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001104
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001104
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMGGhq0f+001116
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMSHhq0f+001116
Adding to the zip: /home/heater/projects/climateprediction.net/oifs_43r3_ps_12200385/ICMUAhq0f+001116
Zipping up the intermediate file: /home/heater/projects/climateprediction.net/oifs_43r3_ps_0741_2014050100_123_983_12200385_0_r98113030_46.zip
Uploading the intermediate file: upload_file_46.zip
  06:38:37 STEP 1129 H=1129:00 +CPU= 50.266
  06:39:34 STEP 1130 H=1130:00 +CPU= 51.520
  06:41:06 STEP 1131 H=1131:00 +CPU= 81.770
  06:42:05 STEP 1132 H=1132:00 +CPU= 53.374
  06:43:01 STEP 1133 H=1133:00 +CPU= 49.484
  06:44:35 STEP 1134 H=1134:00 +CPU= 82.537
  06:46:19 STEP 1135 H=1135:00 +CPU= 51.950
  06:47:20 STEP 1136 H=1136:00 +CPU= 54.021
  09:55:02 STEP 1137 H=1137:00 +CPU=107.030
  09:56:04 STEP 1138 H=1138:00 +CPU= 54.556
  09:57:03 STEP 1139 H=1139:00 +CPU= 52.443
  09:58:47 STEP 1140 H=1140:00 +CPU= 90.923
Moving to projects directory: /home/heater/slots/2/ICMGGhq0f+001140
Moving to projects directory: /home/heater/slots/2/ICMSHhq0f+001140
Moving to projects directory: /home/heater/slots/2/ICMUAhq0f+001140
  09:59:45 STEP 1141 H=1141:00 +CPU= 51.639
  10:00:45 STEP 1142 H=1142:00 +CPU= 53.082
  10:02:19 STEP 1143 H=1143:00 +CPU= 82.276
  10:03:21 STEP 1144 H=1144:00 +CPU= 54.293
  10:04:21 STEP 1145 H=1145:00 +CPU= 51.985
  10:05:53 STEP 1146 H=1146:00 +CPU= 81.755
  10:06:52 STEP 1147 H=1147:00 +CPU= 43.001
  10:07:45 STEP 1148 H=1148:00 +CPU= 47.176
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.950] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.973] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.973] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.978] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x3e900000000], nsigs = 1
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.978] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.978] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.989] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56599.994] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.007] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.007] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100853:1674929333:56600.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                   VDFEXCU 
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100854:1674929334:56601.022] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:BLACKHEATER:1:1:94:94] [20230128:100854:1674929334:56601.022] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FCB6E0  vdfexcu_                  338  vdfexcu.F90
oifs_43r3_model.e  0000000000FAB617  vdfmain_                  981  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:44:13 1536052 22286608 oifs_43r3_ps_0741_2014050100_123_983_12200385_0 3,628,800 60,556 0.0167
09 May 2023 16:44:03 1536052 22286608 oifs_43r3_ps_0741_2014050100_123_983_12200385_0 2,764,800 45,985 0.0166
09 May 2023 16:43:28 1536052 22286608 oifs_43r3_ps_0741_2014050100_123_983_12200385_0 1,728,000 27,674 0.0160
09 May 2023 16:43:19 1536052 22286608 oifs_43r3_ps_0741_2014050100_123_983_12200385_0 864,000 13,960 0.0162


©2024 climateprediction.net