climateprediction.net home page
Task 22271933

Task 22271933

Name oifs_43r3_ps_0913_2000050100_123_969_12186557_0
Workunit 12186557
Created 22 Dec 2022, 22:40:33 UTC
Sent 30 Dec 2022, 14:24:21 UTC
Report deadline 29 Jan 2023, 14:24:21 UTC
Received 30 Dec 2022, 19:59:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537364
Run time 4 hours 35 min 15 sec
CPU time 4 hours 33 min 14 sec
Validate state Invalid
Credit 4,838.40
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.30 MB
Peak swap size 4,974.13 MB
Peak disk usage 1,840.27 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ar/lib/boinc-client/slots/4/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000636
  09:03:19 STEP  637 H= 637:00 +CPU= 11.937
  09:03:31 STEP  638 H= 638:00 +CPU= 12.340
  09:03:50 STEP  639 H= 639:00 +CPU= 18.539
  09:04:02 STEP  640 H= 640:00 +CPU= 12.204
  09:04:14 STEP  641 H= 641:00 +CPU= 11.761
  09:04:33 STEP  642 H= 642:00 +CPU= 18.816
  09:04:45 STEP  643 H= 643:00 +CPU= 11.953
  09:04:57 STEP  644 H= 644:00 +CPU= 12.212
  09:05:16 STEP  645 H= 645:00 +CPU= 18.747
  09:05:28 STEP  646 H= 646:00 +CPU= 12.185
  09:05:40 STEP  647 H= 647:00 +CPU= 12.509
  09:06:03 STEP  648 H= 648:00 +CPU= 21.980
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_26.zip
Uploading the intermediate file: upload_file_26.zip
  09:06:15 STEP  649 H= 649:00 +CPU= 11.972
  09:06:28 STEP  650 H= 650:00 +CPU= 12.323
  09:06:46 STEP  651 H= 651:00 +CPU= 18.316
  09:06:58 STEP  652 H= 652:00 +CPU= 12.378
  09:07:10 STEP  653 H= 653:00 +CPU= 11.783
  09:07:29 STEP  654 H= 654:00 +CPU= 18.993
  09:07:41 STEP  655 H= 655:00 +CPU= 11.834
  09:07:53 STEP  656 H= 656:00 +CPU= 12.162
  09:08:12 STEP  657 H= 657:00 +CPU= 18.440
  09:08:24 STEP  658 H= 658:00 +CPU= 12.132
  09:08:36 STEP  659 H= 659:00 +CPU= 11.971
  09:08:57 STEP  660 H= 660:00 +CPU= 20.547
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000660
  09:09:09 STEP  661 H= 661:00 +CPU= 12.003
  09:09:21 STEP  662 H= 662:00 +CPU= 12.212
  09:09:39 STEP  663 H= 663:00 +CPU= 18.368
  09:09:52 STEP  664 H= 664:00 +CPU= 12.232
  09:10:04 STEP  665 H= 665:00 +CPU= 12.022
  09:10:23 STEP  666 H= 666:00 +CPU= 18.948
  09:10:35 STEP  667 H= 667:00 +CPU= 11.894
  09:10:47 STEP  668 H= 668:00 +CPU= 12.184
  09:11:05 STEP  669 H= 669:00 +CPU= 18.511
  09:11:18 STEP  670 H= 670:00 +CPU= 12.220
  09:11:30 STEP  671 H= 671:00 +CPU= 12.327
  09:11:52 STEP  672 H= 672:00 +CPU= 21.871
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_27.zip
Uploading the intermediate file: upload_file_27.zip
  09:12:04 STEP  673 H= 673:00 +CPU= 11.787
  09:12:16 STEP  674 H= 674:00 +CPU= 12.127
  09:12:35 STEP  675 H= 675:00 +CPU= 18.536
  09:12:47 STEP  676 H= 676:00 +CPU= 12.153
  09:12:59 STEP  677 H= 677:00 +CPU= 11.972
  09:13:18 STEP  678 H= 678:00 +CPU= 19.253
  09:13:30 STEP  679 H= 679:00 +CPU= 12.139
  09:13:43 STEP  680 H= 680:00 +CPU= 12.319
  09:14:01 STEP  681 H= 681:00 +CPU= 18.689
  09:14:13 STEP  682 H= 682:00 +CPU= 12.092
  09:14:26 STEP  683 H= 683:00 +CPU= 11.999
  09:14:46 STEP  684 H= 684:00 +CPU= 20.258
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000684
  09:14:58 STEP  685 H= 685:00 +CPU= 11.695
  09:15:10 STEP  686 H= 686:00 +CPU= 12.340
  09:15:29 STEP  687 H= 687:00 +CPU= 18.700
  09:15:41 STEP  688 H= 688:00 +CPU= 12.256
  09:15:53 STEP  689 H= 689:00 +CPU= 11.830
  09:16:12 STEP  690 H= 690:00 +CPU= 18.943
  09:16:24 STEP  691 H= 691:00 +CPU= 11.786
  09:16:36 STEP  692 H= 692:00 +CPU= 12.487
  09:16:55 STEP  693 H= 693:00 +CPU= 18.549
  09:17:07 STEP  694 H= 694:00 +CPU= 12.367
  09:17:19 STEP  695 H= 695:00 +CPU= 12.251
  09:17:42 STEP  696 H= 696:00 +CPU= 21.990
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_28.zip
Uploading the intermediate file: upload_file_28.zip
  09:17:54 STEP  697 H= 697:00 +CPU= 12.067
  09:18:06 STEP  698 H= 698:00 +CPU= 12.211
  09:18:24 STEP  699 H= 699:00 +CPU= 18.371
  09:18:37 STEP  700 H= 700:00 +CPU= 12.175
  09:18:48 STEP  701 H= 701:00 +CPU= 11.874
  09:19:08 STEP  702 H= 702:00 +CPU= 19.253
  09:19:20 STEP  703 H= 703:00 +CPU= 11.952
  09:19:32 STEP  704 H= 704:00 +CPU= 12.279
  09:19:51 STEP  705 H= 705:00 +CPU= 18.519
  09:20:03 STEP  706 H= 706:00 +CPU= 12.248
  09:20:15 STEP  707 H= 707:00 +CPU= 12.118
  09:20:36 STEP  708 H= 708:00 +CPU= 20.573
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000708
  09:20:48 STEP  709 H= 709:00 +CPU= 12.049
  09:21:00 STEP  710 H= 710:00 +CPU= 12.311
  09:21:19 STEP  711 H= 711:00 +CPU= 18.492
  09:21:31 STEP  712 H= 712:00 +CPU= 12.291
  09:21:43 STEP  713 H= 713:00 +CPU= 11.859
  09:22:02 STEP  714 H= 714:00 +CPU= 19.056
  09:22:14 STEP  715 H= 715:00 +CPU= 11.671
  09:22:26 STEP  716 H= 716:00 +CPU= 12.207
  09:22:45 STEP  717 H= 717:00 +CPU= 18.529
  09:22:57 STEP  718 H= 718:00 +CPU= 12.273
  09:23:09 STEP  719 H= 719:00 +CPU= 12.134
  09:23:31 STEP  720 H= 720:00 +CPU= 21.695
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_29.zip
Uploading the intermediate file: upload_file_29.zip
  09:23:43 STEP  721 H= 721:00 +CPU= 11.892
  09:23:55 STEP  722 H= 722:00 +CPU= 11.994
Uploading trickle at timestep: 2592000
  09:24:13 STEP  723 H= 723:00 +CPU= 18.441
  09:24:25 STEP  724 H= 724:00 +CPU= 12.136
  09:24:37 STEP  725 H= 725:00 +CPU= 11.931
  09:24:56 STEP  726 H= 726:00 +CPU= 18.765
  09:25:08 STEP  727 H= 727:00 +CPU= 12.000
  09:25:21 STEP  728 H= 728:00 +CPU= 12.273
  09:25:39 STEP  729 H= 729:00 +CPU= 18.546
  09:25:51 STEP  730 H= 730:00 +CPU= 11.955
  09:26:03 STEP  731 H= 731:00 +CPU= 11.755
  09:26:23 STEP  732 H= 732:00 +CPU= 20.384
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000732
  09:26:35 STEP  733 H= 733:00 +CPU= 11.932
  09:26:47 STEP  734 H= 734:00 +CPU= 11.956
  09:27:06 STEP  735 H= 735:00 +CPU= 18.626
  09:27:18 STEP  736 H= 736:00 +CPU= 12.169
  09:27:30 STEP  737 H= 737:00 +CPU= 11.715
  09:27:49 STEP  738 H= 738:00 +CPU= 18.867
  09:28:01 STEP  739 H= 739:00 +CPU= 11.629
  09:28:13 STEP  740 H= 740:00 +CPU= 12.157
  09:28:31 STEP  741 H= 741:00 +CPU= 18.161
  09:28:43 STEP  742 H= 742:00 +CPU= 12.113
  09:28:55 STEP  743 H= 743:00 +CPU= 12.095
  09:29:17 STEP  744 H= 744:00 +CPU= 21.911
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_30.zip
Uploading the intermediate file: upload_file_30.zip
  09:29:29 STEP  745 H= 745:00 +CPU= 12.021
  09:29:42 STEP  746 H= 746:00 +CPU= 12.217
  09:30:00 STEP  747 H= 747:00 +CPU= 18.366
  09:30:12 STEP  748 H= 748:00 +CPU= 12.001
  09:30:24 STEP  749 H= 749:00 +CPU= 11.867
  09:30:43 STEP  750 H= 750:00 +CPU= 18.669
  09:30:54 STEP  751 H= 751:00 +CPU= 11.790
  09:31:07 STEP  752 H= 752:00 +CPU= 12.203
  09:31:25 STEP  753 H= 753:00 +CPU= 18.408
  09:31:38 STEP  754 H= 754:00 +CPU= 12.377
  09:31:49 STEP  755 H= 755:00 +CPU= 11.693
  09:32:10 STEP  756 H= 756:00 +CPU= 20.591
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000756
  09:32:22 STEP  757 H= 757:00 +CPU= 11.900
  09:32:34 STEP  758 H= 758:00 +CPU= 12.158
  09:32:53 STEP  759 H= 759:00 +CPU= 18.538
  09:33:05 STEP  760 H= 760:00 +CPU= 12.150
  09:33:17 STEP  761 H= 761:00 +CPU= 11.799
  09:33:36 STEP  762 H= 762:00 +CPU= 18.840
  09:33:47 STEP  763 H= 763:00 +CPU= 11.698
  09:33:59 STEP  764 H= 764:00 +CPU= 12.062
  09:34:18 STEP  765 H= 765:00 +CPU= 18.226
  09:34:30 STEP  766 H= 766:00 +CPU= 11.985
  09:34:42 STEP  767 H= 767:00 +CPU= 12.046
  09:35:04 STEP  768 H= 768:00 +CPU= 21.824
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_31.zip
Uploading the intermediate file: upload_file_31.zip
  09:35:16 STEP  769 H= 769:00 +CPU= 11.825
  09:35:28 STEP  770 H= 770:00 +CPU= 12.030
  09:35:46 STEP  771 H= 771:00 +CPU= 18.366
  09:35:58 STEP  772 H= 772:00 +CPU= 12.013
  09:36:10 STEP  773 H= 773:00 +CPU= 11.786
  09:36:29 STEP  774 H= 774:00 +CPU= 19.050
  09:36:41 STEP  775 H= 775:00 +CPU= 11.733
  09:36:53 STEP  776 H= 776:00 +CPU= 12.037
  09:37:11 STEP  777 H= 777:00 +CPU= 18.559
  09:37:24 STEP  778 H= 778:00 +CPU= 12.099
  09:37:35 STEP  779 H= 779:00 +CPU= 11.713
  09:37:56 STEP  780 H= 780:00 +CPU= 20.130
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000780
  09:38:07 STEP  781 H= 781:00 +CPU= 11.747
  09:38:19 STEP  782 H= 782:00 +CPU= 12.026
  09:38:38 STEP  783 H= 783:00 +CPU= 18.534
  09:38:50 STEP  784 H= 784:00 +CPU= 12.253
  09:39:02 STEP  785 H= 785:00 +CPU= 11.852
  09:39:21 STEP  786 H= 786:00 +CPU= 19.023
  09:39:33 STEP  787 H= 787:00 +CPU= 11.790
  09:39:45 STEP  788 H= 788:00 +CPU= 12.085
  09:40:04 STEP  789 H= 789:00 +CPU= 18.318
  09:40:16 STEP  790 H= 790:00 +CPU= 12.087
  09:40:28 STEP  791 H= 791:00 +CPU= 12.058
  09:40:49 STEP  792 H= 792:00 +CPU= 21.615
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_32.zip
Uploading the intermediate file: upload_file_32.zip
  09:41:01 STEP  793 H= 793:00 +CPU= 11.764
  09:41:13 STEP  794 H= 794:00 +CPU= 12.079
  09:41:32 STEP  795 H= 795:00 +CPU= 18.561
  09:41:44 STEP  796 H= 796:00 +CPU= 12.170
  09:41:56 STEP  797 H= 797:00 +CPU= 11.656
  09:42:15 STEP  798 H= 798:00 +CPU= 18.786
  09:42:27 STEP  799 H= 799:00 +CPU= 11.825
  09:42:39 STEP  800 H= 800:00 +CPU= 12.090
  09:42:57 STEP  801 H= 801:00 +CPU= 18.600
  09:43:10 STEP  802 H= 802:00 +CPU= 12.197
  09:43:21 STEP  803 H= 803:00 +CPU= 11.720
  09:43:42 STEP  804 H= 804:00 +CPU= 20.553
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000804
  09:43:54 STEP  805 H= 805:00 +CPU= 11.886
  09:44:06 STEP  806 H= 806:00 +CPU= 12.176
  09:44:25 STEP  807 H= 807:00 +CPU= 18.486
  09:44:37 STEP  808 H= 808:00 +CPU= 12.021
  09:44:49 STEP  809 H= 809:00 +CPU= 11.836
  09:45:07 STEP  810 H= 810:00 +CPU= 18.637
  09:45:19 STEP  811 H= 811:00 +CPU= 11.741
  09:45:31 STEP  812 H= 812:00 +CPU= 12.189
  09:45:50 STEP  813 H= 813:00 +CPU= 18.249
  09:46:02 STEP  814 H= 814:00 +CPU= 12.189
  09:46:14 STEP  815 H= 815:00 +CPU= 12.283
  09:46:36 STEP  816 H= 816:00 +CPU= 21.908
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_33.zip
Uploading the intermediate file: upload_file_33.zip
  09:46:48 STEP  817 H= 817:00 +CPU= 11.965
  09:47:00 STEP  818 H= 818:00 +CPU= 12.178
  09:47:19 STEP  819 H= 819:00 +CPU= 18.410
  09:47:31 STEP  820 H= 820:00 +CPU= 11.979
  09:47:43 STEP  821 H= 821:00 +CPU= 11.822
  09:48:02 STEP  822 H= 822:00 +CPU= 18.797
  09:48:13 STEP  823 H= 823:00 +CPU= 11.751
  09:48:25 STEP  824 H= 824:00 +CPU= 12.088
  09:48:44 STEP  825 H= 825:00 +CPU= 18.545
  09:48:56 STEP  826 H= 826:00 +CPU= 12.127
  09:49:08 STEP  827 H= 827:00 +CPU= 11.727
  09:49:28 STEP  828 H= 828:00 +CPU= 20.346
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000828
  09:49:40 STEP  829 H= 829:00 +CPU= 11.711
  09:49:52 STEP  830 H= 830:00 +CPU= 12.081
  09:50:11 STEP  831 H= 831:00 +CPU= 18.520
  09:50:23 STEP  832 H= 832:00 +CPU= 11.941
  09:50:34 STEP  833 H= 833:00 +CPU= 11.687
  09:50:53 STEP  834 H= 834:00 +CPU= 18.824
  09:51:05 STEP  835 H= 835:00 +CPU= 11.942
  09:51:17 STEP  836 H= 836:00 +CPU= 12.014
  09:51:36 STEP  837 H= 837:00 +CPU= 18.506
  09:51:48 STEP  838 H= 838:00 +CPU= 12.017
  09:52:00 STEP  839 H= 839:00 +CPU= 12.045
  09:52:22 STEP  840 H= 840:00 +CPU= 21.782
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_34.zip
Uploading the intermediate file: upload_file_34.zip
  09:52:34 STEP  841 H= 841:00 +CPU= 11.835
  09:52:46 STEP  842 H= 842:00 +CPU= 12.092
  09:53:04 STEP  843 H= 843:00 +CPU= 18.348
  09:53:17 STEP  844 H= 844:00 +CPU= 12.091
  09:53:28 STEP  845 H= 845:00 +CPU= 11.607
  09:53:47 STEP  846 H= 846:00 +CPU= 18.732
  09:53:59 STEP  847 H= 847:00 +CPU= 11.745
  09:54:11 STEP  848 H= 848:00 +CPU= 12.049
  09:54:29 STEP  849 H= 849:00 +CPU= 18.554
  09:54:42 STEP  850 H= 850:00 +CPU= 12.085
  09:54:53 STEP  851 H= 851:00 +CPU= 11.695
  09:55:14 STEP  852 H= 852:00 +CPU= 20.464
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000852
  09:55:25 STEP  853 H= 853:00 +CPU= 11.598
  09:55:38 STEP  854 H= 854:00 +CPU= 12.174
  09:55:56 STEP  855 H= 855:00 +CPU= 18.183
  09:56:08 STEP  856 H= 856:00 +CPU= 12.063
  09:56:20 STEP  857 H= 857:00 +CPU= 11.643
  09:56:39 STEP  858 H= 858:00 +CPU= 18.813
  09:56:50 STEP  859 H= 859:00 +CPU= 11.774
  09:57:02 STEP  860 H= 860:00 +CPU= 12.093
  09:57:21 STEP  861 H= 861:00 +CPU= 18.278
  09:57:33 STEP  862 H= 862:00 +CPU= 12.134
  09:57:45 STEP  863 H= 863:00 +CPU= 12.081
  09:58:07 STEP  864 H= 864:00 +CPU= 21.807
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_35.zip
Uploading the intermediate file: upload_file_35.zip
  09:58:19 STEP  865 H= 865:00 +CPU= 11.800
  09:58:31 STEP  866 H= 866:00 +CPU= 12.231
  09:58:50 STEP  867 H= 867:00 +CPU= 18.580
  09:59:02 STEP  868 H= 868:00 +CPU= 12.238
  09:59:14 STEP  869 H= 869:00 +CPU= 11.678
  09:59:33 STEP  870 H= 870:00 +CPU= 18.939
  09:59:45 STEP  871 H= 871:00 +CPU= 11.909
  09:59:57 STEP  872 H= 872:00 +CPU= 12.238
  10:00:15 STEP  873 H= 873:00 +CPU= 18.481
  10:00:27 STEP  874 H= 874:00 +CPU= 12.013
  10:00:39 STEP  875 H= 875:00 +CPU= 11.893
  10:01:00 STEP  876 H= 876:00 +CPU= 20.288
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000876
  10:01:11 STEP  877 H= 877:00 +CPU= 11.664
  10:01:24 STEP  878 H= 878:00 +CPU= 12.050
  10:01:42 STEP  879 H= 879:00 +CPU= 18.429
  10:01:54 STEP  880 H= 880:00 +CPU= 11.989
  10:02:06 STEP  881 H= 881:00 +CPU= 11.788
  10:02:25 STEP  882 H= 882:00 +CPU= 18.771
  10:02:36 STEP  883 H= 883:00 +CPU= 11.715
  10:02:49 STEP  884 H= 884:00 +CPU= 12.096
  10:03:07 STEP  885 H= 885:00 +CPU= 18.463
  10:03:19 STEP  886 H= 886:00 +CPU= 12.085
  10:03:31 STEP  887 H= 887:00 +CPU= 12.022
  10:03:53 STEP  888 H= 888:00 +CPU= 21.988
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_36.zip
Uploading the intermediate file: upload_file_36.zip
  10:04:05 STEP  889 H= 889:00 +CPU= 11.603
  10:04:17 STEP  890 H= 890:00 +CPU= 12.069
  10:04:36 STEP  891 H= 891:00 +CPU= 18.432
  10:04:48 STEP  892 H= 892:00 +CPU= 11.976
  10:04:59 STEP  893 H= 893:00 +CPU= 11.805
  10:05:18 STEP  894 H= 894:00 +CPU= 18.987
  10:05:30 STEP  895 H= 895:00 +CPU= 11.762
  10:05:42 STEP  896 H= 896:00 +CPU= 12.188
  10:06:01 STEP  897 H= 897:00 +CPU= 18.417
  10:06:13 STEP  898 H= 898:00 +CPU= 12.144
  10:06:25 STEP  899 H= 899:00 +CPU= 11.785
  10:06:45 STEP  900 H= 900:00 +CPU= 20.372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000900
  10:06:57 STEP  901 H= 901:00 +CPU= 11.664
  10:07:09 STEP  902 H= 902:00 +CPU= 12.110
  10:07:28 STEP  903 H= 903:00 +CPU= 18.394
  10:07:40 STEP  904 H= 904:00 +CPU= 12.213
  10:07:52 STEP  905 H= 905:00 +CPU= 11.858
  10:08:11 STEP  906 H= 906:00 +CPU= 18.823
  10:08:22 STEP  907 H= 907:00 +CPU= 11.665
  10:08:35 STEP  908 H= 908:00 +CPU= 12.114
  10:08:53 STEP  909 H= 909:00 +CPU= 18.481
  10:09:05 STEP  910 H= 910:00 +CPU= 12.240
  10:09:18 STEP  911 H= 911:00 +CPU= 12.162
  10:09:39 STEP  912 H= 912:00 +CPU= 21.732
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_37.zip
Uploading the intermediate file: upload_file_37.zip
  10:09:51 STEP  913 H= 913:00 +CPU= 11.784
  10:10:03 STEP  914 H= 914:00 +CPU= 12.136
  10:10:22 STEP  915 H= 915:00 +CPU= 18.430
  10:10:34 STEP  916 H= 916:00 +CPU= 12.099
  10:10:46 STEP  917 H= 917:00 +CPU= 11.676
  10:11:05 STEP  918 H= 918:00 +CPU= 19.121
  10:11:16 STEP  919 H= 919:00 +CPU= 11.640
  10:11:29 STEP  920 H= 920:00 +CPU= 12.194
  10:11:47 STEP  921 H= 921:00 +CPU= 18.409
  10:11:59 STEP  922 H= 922:00 +CPU= 12.088
  10:12:11 STEP  923 H= 923:00 +CPU= 11.853
  10:12:32 STEP  924 H= 924:00 +CPU= 20.597
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000924
  10:12:45 STEP  925 H= 925:00 +CPU= 12.624
  10:12:57 STEP  926 H= 926:00 +CPU= 12.421
  10:13:17 STEP  927 H= 927:00 +CPU= 19.024
  10:13:29 STEP  928 H= 928:00 +CPU= 12.397
  10:13:41 STEP  929 H= 929:00 +CPU= 11.829
  10:14:01 STEP  930 H= 930:00 +CPU= 19.206
  10:14:13 STEP  931 H= 931:00 +CPU= 11.925
  10:14:25 STEP  932 H= 932:00 +CPU= 12.199
  10:14:44 STEP  933 H= 933:00 +CPU= 18.788
  10:14:56 STEP  934 H= 934:00 +CPU= 12.137
  10:15:09 STEP  935 H= 935:00 +CPU= 12.390
  10:15:31 STEP  936 H= 936:00 +CPU= 22.193
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_38.zip
Uploading the intermediate file: upload_file_38.zip
  10:15:43 STEP  937 H= 937:00 +CPU= 12.144
  10:15:56 STEP  938 H= 938:00 +CPU= 12.130
  10:16:14 STEP  939 H= 939:00 +CPU= 18.637
  10:16:27 STEP  940 H= 940:00 +CPU= 12.156
  10:16:39 STEP  941 H= 941:00 +CPU= 11.933
  10:16:58 STEP  942 H= 942:00 +CPU= 18.861
  10:17:10 STEP  943 H= 943:00 +CPU= 12.024
  10:17:22 STEP  944 H= 944:00 +CPU= 12.231
  10:17:42 STEP  945 H= 945:00 +CPU= 19.443
  10:17:55 STEP  946 H= 946:00 +CPU= 12.825
  10:18:07 STEP  947 H= 947:00 +CPU= 11.852
  10:18:27 STEP  948 H= 948:00 +CPU= 20.458
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000948
  10:18:39 STEP  949 H= 949:00 +CPU= 11.812
  10:18:51 STEP  950 H= 950:00 +CPU= 12.054
  10:19:10 STEP  951 H= 951:00 +CPU= 18.378
  10:19:22 STEP  952 H= 952:00 +CPU= 12.150
  10:19:34 STEP  953 H= 953:00 +CPU= 11.835
  10:19:53 STEP  954 H= 954:00 +CPU= 19.017
  10:20:05 STEP  955 H= 955:00 +CPU= 11.883
  10:20:17 STEP  956 H= 956:00 +CPU= 12.261
  10:20:36 STEP  957 H= 957:00 +CPU= 18.296
  10:20:48 STEP  958 H= 958:00 +CPU= 12.107
  10:21:00 STEP  959 H= 959:00 +CPU= 12.110
  10:21:22 STEP  960 H= 960:00 +CPU= 21.808
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_39.zip
Uploading the intermediate file: upload_file_39.zip
  10:21:34 STEP  961 H= 961:00 +CPU= 12.011
Uploading trickle at timestep: 3456000
  10:21:47 STEP  962 H= 962:00 +CPU= 12.537
  10:22:06 STEP  963 H= 963:00 +CPU= 18.895
  10:22:18 STEP  964 H= 964:00 +CPU= 12.657
  10:22:30 STEP  965 H= 965:00 +CPU= 11.772
  10:22:49 STEP  966 H= 966:00 +CPU= 18.824
  10:23:01 STEP  967 H= 967:00 +CPU= 11.696
  10:23:13 STEP  968 H= 968:00 +CPU= 11.924
  10:23:31 STEP  969 H= 969:00 +CPU= 18.448
  10:23:43 STEP  970 H= 970:00 +CPU= 11.978
  10:23:55 STEP  971 H= 971:00 +CPU= 11.757
  10:24:16 STEP  972 H= 972:00 +CPU= 20.562
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000972
  10:24:28 STEP  973 H= 973:00 +CPU= 11.779
  10:24:40 STEP  974 H= 974:00 +CPU= 12.076
  10:24:58 STEP  975 H= 975:00 +CPU= 18.475
  10:25:10 STEP  976 H= 976:00 +CPU= 12.021
  10:25:22 STEP  977 H= 977:00 +CPU= 11.699
  10:25:41 STEP  978 H= 978:00 +CPU= 19.064
  10:25:53 STEP  979 H= 979:00 +CPU= 11.788
  10:26:05 STEP  980 H= 980:00 +CPU= 12.160
  10:26:23 STEP  981 H= 981:00 +CPU= 18.295
  10:26:36 STEP  982 H= 982:00 +CPU= 12.036
  10:26:48 STEP  983 H= 983:00 +CPU= 12.082
  10:27:10 STEP  984 H= 984:00 +CPU= 21.638
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_40.zip
Uploading the intermediate file: upload_file_40.zip
  10:27:22 STEP  985 H= 985:00 +CPU= 11.712
  10:27:34 STEP  986 H= 986:00 +CPU= 12.025
  10:27:52 STEP  987 H= 987:00 +CPU= 18.390
  10:28:04 STEP  988 H= 988:00 +CPU= 12.192
  10:28:16 STEP  989 H= 989:00 +CPU= 11.752
  10:28:35 STEP  990 H= 990:00 +CPU= 19.219
  10:28:47 STEP  991 H= 991:00 +CPU= 11.733
  10:28:59 STEP  992 H= 992:00 +CPU= 12.133
  10:29:18 STEP  993 H= 993:00 +CPU= 18.594
  10:29:30 STEP  994 H= 994:00 +CPU= 12.235
  10:29:42 STEP  995 H= 995:00 +CPU= 11.800
  10:30:03 STEP  996 H= 996:00 +CPU= 20.449
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000996
  10:30:14 STEP  997 H= 997:00 +CPU= 11.830
  10:30:27 STEP  998 H= 998:00 +CPU= 12.168
  10:30:45 STEP  999 H= 999:00 +CPU= 18.275
  10:30:57 STEP 1000 H=1000:00 +CPU= 12.323
  10:31:09 STEP 1001 H=1001:00 +CPU= 11.747
  10:31:29 STEP 1002 H=1002:00 +CPU= 19.304
  10:31:41 STEP 1003 H=1003:00 +CPU= 11.847
  10:31:53 STEP 1004 H=1004:00 +CPU= 12.350
  10:32:11 STEP 1005 H=1005:00 +CPU= 18.393
  10:32:23 STEP 1006 H=1006:00 +CPU= 11.984
  10:32:36 STEP 1007 H=1007:00 +CPU= 12.269
  10:32:58 STEP 1008 H=1008:00 +CPU= 21.872
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_41.zip
Uploading the intermediate file: upload_file_41.zip
  10:33:10 STEP 1009 H=1009:00 +CPU= 12.049
  10:33:22 STEP 1010 H=1010:00 +CPU= 12.034
  10:33:41 STEP 1011 H=1011:00 +CPU= 18.740
  10:33:53 STEP 1012 H=1012:00 +CPU= 11.927
  10:34:04 STEP 1013 H=1013:00 +CPU= 11.661
  10:34:23 STEP 1014 H=1014:00 +CPU= 18.995
  10:34:35 STEP 1015 H=1015:00 +CPU= 11.861
  10:34:47 STEP 1016 H=1016:00 +CPU= 12.150
  10:35:06 STEP 1017 H=1017:00 +CPU= 18.456
  10:35:18 STEP 1018 H=1018:00 +CPU= 12.153
  10:35:30 STEP 1019 H=1019:00 +CPU= 11.730
  10:35:50 STEP 1020 H=1020:00 +CPU= 20.230
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001020
  10:36:02 STEP 1021 H=1021:00 +CPU= 11.688
  10:36:14 STEP 1022 H=1022:00 +CPU= 11.916
  10:36:32 STEP 1023 H=1023:00 +CPU= 18.287
  10:36:44 STEP 1024 H=1024:00 +CPU= 12.167
  10:36:56 STEP 1025 H=1025:00 +CPU= 11.921
  10:37:15 STEP 1026 H=1026:00 +CPU= 18.821
  10:37:27 STEP 1027 H=1027:00 +CPU= 11.739
  10:37:39 STEP 1028 H=1028:00 +CPU= 12.082
  10:37:57 STEP 1029 H=1029:00 +CPU= 18.248
  10:38:09 STEP 1030 H=1030:00 +CPU= 12.047
  10:38:22 STEP 1031 H=1031:00 +CPU= 12.144
  10:38:43 STEP 1032 H=1032:00 +CPU= 21.520
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_42.zip
Uploading the intermediate file: upload_file_42.zip
  10:38:55 STEP 1033 H=1033:00 +CPU= 11.691
  10:39:07 STEP 1034 H=1034:00 +CPU= 12.163
  10:39:26 STEP 1035 H=1035:00 +CPU= 18.438
  10:39:38 STEP 1036 H=1036:00 +CPU= 12.169
  10:39:49 STEP 1037 H=1037:00 +CPU= 11.520
  10:40:08 STEP 1038 H=1038:00 +CPU= 18.580
  10:40:20 STEP 1039 H=1039:00 +CPU= 11.807
  10:40:32 STEP 1040 H=1040:00 +CPU= 12.235
  10:40:51 STEP 1041 H=1041:00 +CPU= 18.334
  10:41:03 STEP 1042 H=1042:00 +CPU= 12.001
  10:41:14 STEP 1043 H=1043:00 +CPU= 11.707
  10:41:35 STEP 1044 H=1044:00 +CPU= 20.358
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001044
  10:41:46 STEP 1045 H=1045:00 +CPU= 11.686
  10:41:58 STEP 1046 H=1046:00 +CPU= 11.940
  10:42:17 STEP 1047 H=1047:00 +CPU= 18.635
  10:42:29 STEP 1048 H=1048:00 +CPU= 12.177
  10:42:41 STEP 1049 H=1049:00 +CPU= 11.836
  10:43:00 STEP 1050 H=1050:00 +CPU= 18.880
  10:43:12 STEP 1051 H=1051:00 +CPU= 11.780
  10:43:24 STEP 1052 H=1052:00 +CPU= 11.876
  10:43:42 STEP 1053 H=1053:00 +CPU= 18.444
  10:43:55 STEP 1054 H=1054:00 +CPU= 12.205
  10:44:07 STEP 1055 H=1055:00 +CPU= 12.149
  10:44:28 STEP 1056 H=1056:00 +CPU= 21.501
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_43.zip
Uploading the intermediate file: upload_file_43.zip
  10:44:40 STEP 1057 H=1057:00 +CPU= 11.659
  10:44:52 STEP 1058 H=1058:00 +CPU= 12.099
  10:45:12 STEP 1059 H=1059:00 +CPU= 19.357
  10:45:24 STEP 1060 H=1060:00 +CPU= 12.140
  10:45:36 STEP 1061 H=1061:00 +CPU= 11.688
  10:45:55 STEP 1062 H=1062:00 +CPU= 18.931
  10:46:06 STEP 1063 H=1063:00 +CPU= 11.585
  10:46:19 STEP 1064 H=1064:00 +CPU= 12.107
  10:46:37 STEP 1065 H=1065:00 +CPU= 18.625
  10:46:49 STEP 1066 H=1066:00 +CPU= 12.130
  10:47:01 STEP 1067 H=1067:00 +CPU= 11.710
  10:47:22 STEP 1068 H=1068:00 +CPU= 20.297
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001068
  10:47:34 STEP 1069 H=1069:00 +CPU= 11.869
  10:47:46 STEP 1070 H=1070:00 +CPU= 12.181
  10:48:04 STEP 1071 H=1071:00 +CPU= 18.169
  10:48:16 STEP 1072 H=1072:00 +CPU= 11.990
  10:48:28 STEP 1073 H=1073:00 +CPU= 11.839
  10:48:47 STEP 1074 H=1074:00 +CPU= 18.852
  10:48:59 STEP 1075 H=1075:00 +CPU= 11.761
  10:49:11 STEP 1076 H=1076:00 +CPU= 12.118
  10:49:29 STEP 1077 H=1077:00 +CPU= 18.452
  10:49:42 STEP 1078 H=1078:00 +CPU= 12.250
  10:49:54 STEP 1079 H=1079:00 +CPU= 12.121
  10:50:16 STEP 1080 H=1080:00 +CPU= 21.753
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_44.zip
Uploading the intermediate file: upload_file_44.zip
  10:50:28 STEP 1081 H=1081:00 +CPU= 11.889
  10:50:40 STEP 1082 H=1082:00 +CPU= 12.112
  10:50:58 STEP 1083 H=1083:00 +CPU= 18.575
  10:51:11 STEP 1084 H=1084:00 +CPU= 12.013
  10:51:22 STEP 1085 H=1085:00 +CPU= 11.761
  10:51:41 STEP 1086 H=1086:00 +CPU= 19.050
  10:51:53 STEP 1087 H=1087:00 +CPU= 11.725
  10:52:05 STEP 1088 H=1088:00 +CPU= 12.228
  10:52:24 STEP 1089 H=1089:00 +CPU= 18.492
  10:52:36 STEP 1090 H=1090:00 +CPU= 12.254
  10:52:48 STEP 1091 H=1091:00 +CPU= 11.764
  10:53:09 STEP 1092 H=1092:00 +CPU= 20.514
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001092
  10:53:21 STEP 1093 H=1093:00 +CPU= 11.962
  10:53:33 STEP 1094 H=1094:00 +CPU= 12.038
  10:53:52 STEP 1095 H=1095:00 +CPU= 18.715
  10:54:04 STEP 1096 H=1096:00 +CPU= 12.287
  10:54:16 STEP 1097 H=1097:00 +CPU= 11.801
  10:54:35 STEP 1098 H=1098:00 +CPU= 19.051
  10:54:47 STEP 1099 H=1099:00 +CPU= 11.792
  10:54:59 STEP 1100 H=1100:00 +CPU= 12.293
  10:55:18 STEP 1101 H=1101:00 +CPU= 18.440
  10:55:30 STEP 1102 H=1102:00 +CPU= 12.241
  10:55:42 STEP 1103 H=1103:00 +CPU= 12.171
  10:56:04 STEP 1104 H=1104:00 +CPU= 21.945
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12186557/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0913_2000050100_123_969_12186557_0_r421286852_45.zip
Uploading the intermediate file: upload_file_45.zip
  10:56:16 STEP 1105 H=1105:00 +CPU= 11.831
  10:56:28 STEP 1106 H=1106:00 +CPU= 12.252
  10:56:47 STEP 1107 H=1107:00 +CPU= 18.527
  10:56:59 STEP 1108 H=1108:00 +CPU= 12.285
  10:57:11 STEP 1109 H=1109:00 +CPU= 11.795
  10:57:30 STEP 1110 H=1110:00 +CPU= 18.872
  10:57:42 STEP 1111 H=1111:00 +CPU= 11.861
  10:57:54 STEP 1112 H=1112:00 +CPU= 12.187
  10:58:13 STEP 1113 H=1113:00 +CPU= 18.515
  10:58:25 STEP 1114 H=1114:00 +CPU= 12.132
 SMILAG TRAJECTORY OUT OF ATM      1 TIMES.
  POINT=           19  MAX ETADOT VERTICAL VEL.=   5.284995519136200E-003
  LON  =    160.875000000000       degrees
  LAT  =   -24.1120250996709       degrees
  10:58:37 STEP 1115 H=1115:00 +CPU= 11.885
 SMILAG TRAJECTORY OUT OF ATM      1 TIMES.
  POINT=            5  MAX ETADOT VERTICAL VEL.=   2.238737482924974E-003
  LON  =    67.5000000000000       degrees
  LAT  =   -75.6998442220113       degrees
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#11 (SIGSEGV) :: 4745MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#11 was caused by address not mapped to object [memaddr=0x14b35c673488], nsigs = 1
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#11, nsigs = 1
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4745 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     2        (1005) 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEB 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARCINB 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 LAITRE_GMV 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105848:1672426728:16450.344] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  LAITRI 
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/4/oifs_43r3_model.exe' : sigcontextptr=0x7ffc2c30d390
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 22) with addr2line-cmd
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/4/oifs_43r3_model.exe' 0x839273 0x839879 0x1dc9090 0xdd20e9 0xddbee4 0xdbec9a 0xd7eba8 0xd7b935 0xd784fd 0xbf9958 0x12e9482 0xbbc1ce 0x57a664 0x55deec 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dc9090] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0xdd20e9] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xddbee4] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xdbec9a] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xd7eba8] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xd7b935] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xd784fd] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xbf9958] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x12e9482] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbbc1ce] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x57a664] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x55deec] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x5471c9] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x546412] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x545e28] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x40708d] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40220f] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x4021a2] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x1dca390] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x40206e] : ??() at ??:0
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.346] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/4/oifs_43r3_model.exe' : sigcontextptr=0x7ffc2c30d2a0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 23) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/4/oifs_43r3_model.exe' 0x839273 0x839879 0x821070 0x1dc9090 0xdd20e9 0xddbee4 0xdbec9a 0xd7eba8 0xd7b935 0xd784fd 0xbf9958 0x12e9482 0xbbc1ce 0x57a664 0x55deec 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x821070] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dc9090] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xdd20e9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xddbee4] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xdbec9a] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xd7eba8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xd7b935] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xd784fd] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbf9958] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12e9482] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbbc1ce] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x57a664] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x55deec] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.354] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#11, nsigs = 1
[EC_DRHOOK:ted:1:1:1122793:1122793] [20221230:105849:1672426729:16451.354] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#11, nsigs = 1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE913A  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DD20E9  laitri_                   313  laitri.F90
oifs_43r3_model.e  0000000000DDBEE4  laitre_gmv_               143  laitre_gmv.F90
oifs_43r3_model.e  0000000000DBEC9A  larcinb_                  415  larcinb.F90
oifs_43r3_model.e  0000000000D7EBA8  lapineb_                  451  lapineb.F90
oifs_43r3_model.e  0000000000D7B935  call_sl_                  363  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:03:12 1537364 22271933 oifs_43r3_ps_0913_2000050100_123_969_12186557_0 3,456,000 14,157 0.0041
09 May 2023 16:03:09 1537364 22271933 oifs_43r3_ps_0913_2000050100_123_969_12186557_0 2,592,000 10,707 0.0041
09 May 2023 16:03:07 1537364 22271933 oifs_43r3_ps_0913_2000050100_123_969_12186557_0 1,728,000 7,224 0.0042
09 May 2023 16:03:07 1537364 22271933 oifs_43r3_ps_0913_2000050100_123_969_12186557_0 1,728,000 7,224 0.0042
09 May 2023 16:03:03 1537364 22271933 oifs_43r3_ps_0913_2000050100_123_969_12186557_0 864,000 3,571 0.0041


©2024 cpdn.org