climateprediction.net home page
Task 22287519

Task 22287519

Name oifs_43r3_ps_0588_2015050100_123_984_12201232_0
Workunit 12201232
Created 24 Dec 2022, 13:10:41 UTC
Sent 21 Jan 2023, 19:37:00 UTC
Report deadline 22 Mar 2023, 19:37:00 UTC
Received 27 Jan 2023, 12:10:25 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1513123
Run time 13 hours 33 min 44 sec
CPU time 13 hours 33 min 44 sec
Validate state Invalid
Credit 4,112.64
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,768.85 MB
Peak swap size 4,978.06 MB
Peak disk usage 3,089.98 MB
Stderr 
<core_client_version>7.16.17</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
 process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:12:26 STEP  747 H= 747:00 +CPU= 69.932
  03:13:22 STEP  748 H= 748:00 +CPU= 46.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:14:26 STEP  749 H= 749:00 +CPU= 46.555
  03:15:59 STEP  750 H= 750:00 +CPU= 66.551
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:16:50 STEP  751 H= 751:00 +CPU= 43.605
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:18:15 STEP  752 H= 752:00 +CPU= 48.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:19:38 STEP  753 H= 753:00 +CPU= 73.232
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:20:40 STEP  754 H= 754:00 +CPU= 49.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:21:42 STEP  755 H= 755:00 +CPU= 47.706
  03:23:14 STEP  756 H= 756:00 +CPU= 82.274
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000756
  03:24:06 STEP  757 H= 757:00 +CPU= 46.617
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:25:04 STEP  758 H= 758:00 +CPU= 46.757
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:26:34 STEP  759 H= 759:00 +CPU= 70.470
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:27:37 STEP  760 H= 760:00 +CPU= 46.361
  03:28:38 STEP  761 H= 761:00 +CPU= 42.668
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:30:08 STEP  762 H= 762:00 +CPU= 71.012
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:31:02 STEP  763 H= 763:00 +CPU= 45.943
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:31:54 STEP  764 H= 764:00 +CPU= 45.834
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:33:11 STEP  765 H= 765:00 +CPU= 66.536
  03:33:53 STEP  766 H= 766:00 +CPU= 40.917
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:34:40 STEP  767 H= 767:00 +CPU= 39.464
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:36:16 STEP  768 H= 768:00 +CPU= 80.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994_31.zip
  03:37:12 STEP  769 H= 769:00 +CPU= 43.058
Uploading the intermediate file: upload_file_31.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:38:13 STEP  770 H= 770:00 +CPU= 50.945
  03:39:55 STEP  771 H= 771:00 +CPU= 87.878
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:41:27 STEP  772 H= 772:00 +CPU= 66.600
  03:42:25 STEP  773 H= 773:00 +CPU= 44.237
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:44:17 STEP  774 H= 774:00 +CPU= 73.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:45:10 STEP  775 H= 775:00 +CPU= 45.300
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:46:20 STEP  776 H= 776:00 +CPU= 51.759
  03:47:35 STEP  777 H= 777:00 +CPU= 67.537
  03:48:28 STEP  778 H= 778:00 +CPU= 48.793
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:49:27 STEP  779 H= 779:00 +CPU= 46.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:51:08 STEP  780 H= 780:00 +CPU= 84.984
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:52:02 STEP  781 H= 781:00 +CPU= 47.035
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:52:56 STEP  782 H= 782:00 +CPU= 45.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:54:16 STEP  783 H= 783:00 +CPU= 68.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:55:08 STEP  784 H= 784:00 +CPU= 44.927
  03:56:00 STEP  785 H= 785:00 +CPU= 45.110
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:57:53 STEP  786 H= 786:00 +CPU= 73.307
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:58:52 STEP  787 H= 787:00 +CPU= 48.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:59:48 STEP  788 H= 788:00 +CPU= 46.164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:01:20 STEP  789 H= 789:00 +CPU= 74.707
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:02:21 STEP  790 H= 790:00 +CPU= 49.864
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:03:21 STEP  791 H= 791:00 +CPU= 47.544
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:06:08 STEP  792 H= 792:00 +CPU= 98.057
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994_32.zip
  04:07:12 STEP  793 H= 793:00 +CPU= 55.631
Uploading the intermediate file: upload_file_32.zip
  04:07:56 STEP  794 H= 794:00 +CPU= 41.226
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:09:37 STEP  795 H= 795:00 +CPU= 70.366
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:10:33 STEP  796 H= 796:00 +CPU= 46.235
  04:11:29 STEP  797 H= 797:00 +CPU= 48.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:12:52 STEP  798 H= 798:00 +CPU= 74.712
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:13:55 STEP  799 H= 799:00 +CPU= 48.777
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:15:04 STEP  800 H= 800:00 +CPU= 50.730
  04:16:44 STEP  801 H= 801:00 +CPU= 85.234
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:17:45 STEP  802 H= 802:00 +CPU= 53.955
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:18:52 STEP  803 H= 803:00 +CPU= 47.748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:20:33 STEP  804 H= 804:00 +CPU= 80.554
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000804
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:21:54 STEP  805 H= 805:00 +CPU= 58.473
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:22:58 STEP  806 H= 806:00 +CPU= 54.365
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:24:45 STEP  807 H= 807:00 +CPU= 78.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:25:52 STEP  808 H= 808:00 +CPU= 51.080
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:26:55 STEP  809 H= 809:00 +CPU= 52.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:28:18 STEP  810 H= 810:00 +CPU= 72.368
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:29:18 STEP  811 H= 811:00 +CPU= 45.247
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:30:18 STEP  812 H= 812:00 +CPU= 46.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:31:46 STEP  813 H= 813:00 +CPU= 71.541
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:33:07 STEP  814 H= 814:00 +CPU= 48.161
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:34:13 STEP  815 H= 815:00 +CPU= 46.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:35:53 STEP  816 H= 816:00 +CPU= 84.848
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994_33.zip
Uploading the intermediate file: upload_file_33.zip
Uploading trickle at timestep: 2937600
  04:36:35 STEP  817 H= 817:00 +CPU= 40.504
  04:37:27 STEP  818 H= 818:00 +CPU= 45.919
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:39:08 STEP  819 H= 819:00 +CPU= 71.825
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:40:11 STEP  820 H= 820:00 +CPU= 49.910
  04:41:01 STEP  821 H= 821:00 +CPU= 44.580
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:42:34 STEP  822 H= 822:00 +CPU= 77.018
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:43:32 STEP  823 H= 823:00 +CPU= 46.699
  04:44:33 STEP  824 H= 824:00 +CPU= 48.627
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:46:01 STEP  825 H= 825:00 +CPU= 76.251
  04:46:54 STEP  826 H= 826:00 +CPU= 47.727
  04:48:17 STEP  827 H= 827:00 +CPU= 42.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:49:52 STEP  828 H= 828:00 +CPU= 86.507
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000828
  04:50:40 STEP  829 H= 829:00 +CPU= 45.176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:51:36 STEP  830 H= 830:00 +CPU= 48.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:53:00 STEP  831 H= 831:00 +CPU= 71.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:53:48 STEP  832 H= 832:00 +CPU= 44.422
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:54:43 STEP  833 H= 833:00 +CPU= 45.133
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:56:24 STEP  834 H= 834:00 +CPU= 73.811
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:57:18 STEP  835 H= 835:00 +CPU= 45.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:58:17 STEP  836 H= 836:00 +CPU= 49.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:59:47 STEP  837 H= 837:00 +CPU= 75.150
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:00:46 STEP  838 H= 838:00 +CPU= 48.782
  05:01:40 STEP  839 H= 839:00 +CPU= 46.776
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:03:27 STEP  840 H= 840:00 +CPU= 83.707
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994_34.zip
Uploading the intermediate file: upload_file_34.zip
  05:04:17 STEP  841 H= 841:00 +CPU= 44.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:05:34 STEP  842 H= 842:00 +CPU= 45.876
  05:07:01 STEP  843 H= 843:00 +CPU= 74.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:07:57 STEP  844 H= 844:00 +CPU= 44.902
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:08:45 STEP  845 H= 845:00 +CPU= 43.430
  05:10:00 STEP  846 H= 846:00 +CPU= 68.647
  05:10:52 STEP  847 H= 847:00 +CPU= 44.862
  05:11:49 STEP  848 H= 848:00 +CPU= 46.526
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:13:25 STEP  849 H= 849:00 +CPU= 76.263
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:14:49 STEP  850 H= 850:00 +CPU= 45.986
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:15:50 STEP  851 H= 851:00 +CPU= 47.374
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:17:48 STEP  852 H= 852:00 +CPU= 81.410
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000852
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:18:46 STEP  853 H= 853:00 +CPU= 45.824
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:19:30 STEP  854 H= 854:00 +CPU= 41.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:20:41 STEP  855 H= 855:00 +CPU= 64.003
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:21:57 STEP  856 H= 856:00 +CPU= 39.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:22:48 STEP  857 H= 857:00 +CPU= 43.181
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:24:12 STEP  858 H= 858:00 +CPU= 71.067
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:24:59 STEP  859 H= 859:00 +CPU= 42.280
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:25:48 STEP  860 H= 860:00 +CPU= 43.476
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:27:12 STEP  861 H= 861:00 +CPU= 69.499
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:28:36 STEP  862 H= 862:00 +CPU= 43.124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:29:38 STEP  863 H= 863:00 +CPU= 46.098
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:31:52 STEP  864 H= 864:00 +CPU= 78.088
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994_35.zip
Uploading the intermediate file: upload_file_35.zip
  05:32:37 STEP  865 H= 865:00 +CPU= 39.999
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:33:24 STEP  866 H= 866:00 +CPU= 39.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:34:38 STEP  867 H= 867:00 +CPU= 62.556
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:35:24 STEP  868 H= 868:00 +CPU= 39.957
  05:36:05 STEP  869 H= 869:00 +CPU= 39.781
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:37:25 STEP  870 H= 870:00 +CPU= 70.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:38:43 STEP  871 H= 871:00 +CPU= 45.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:39:32 STEP  872 H= 872:00 +CPU= 45.024
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:40:53 STEP  873 H= 873:00 +CPU= 68.635
  05:41:51 STEP  874 H= 874:00 +CPU= 47.370
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:43:12 STEP  875 H= 875:00 +CPU= 44.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:44:50 STEP  876 H= 876:00 +CPU= 83.636
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000876
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:46:09 STEP  877 H= 877:00 +CPU= 45.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:47:07 STEP  878 H= 878:00 +CPU= 47.837
  05:48:29 STEP  879 H= 879:00 +CPU= 67.660
  05:49:23 STEP  880 H= 880:00 +CPU= 49.359
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:50:19 STEP  881 H= 881:00 +CPU= 42.309
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:51:50 STEP  882 H= 882:00 +CPU= 71.076
  05:52:49 STEP  883 H= 883:00 +CPU= 45.434
  05:53:46 STEP  884 H= 884:00 +CPU= 49.652
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:55:12 STEP  885 H= 885:00 +CPU= 70.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:56:46 STEP  886 H= 886:00 +CPU= 42.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:58:45 STEP  887 H= 887:00 +CPU= 53.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:00:32 STEP  888 H= 888:00 +CPU= 78.523
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994_36.zip
Uploading the intermediate file: upload_file_36.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:01:25 STEP  889 H= 889:00 +CPU= 44.246
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:02:25 STEP  890 H= 890:00 +CPU= 45.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:51 STEP  891 H= 891:00 +CPU= 73.876
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:04:45 STEP  892 H= 892:00 +CPU= 46.447
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:05:37 STEP  893 H= 893:00 +CPU= 44.923
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:07:09 STEP  894 H= 894:00 +CPU= 74.020
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:08:01 STEP  895 H= 895:00 +CPU= 43.902
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:08:52 STEP  896 H= 896:00 +CPU= 42.440
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:10:13 STEP  897 H= 897:00 +CPU= 70.043
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:11:34 STEP  898 H= 898:00 +CPU= 43.773
  06:12:30 STEP  899 H= 899:00 +CPU= 48.595
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:15:39 STEP  900 H= 900:00 +CPU= 79.064
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000900
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:16:32 STEP  901 H= 901:00 +CPU= 44.334
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:17:33 STEP  902 H= 902:00 +CPU= 48.983
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:19:29 STEP  903 H= 903:00 +CPU= 87.579
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:20:56 STEP  904 H= 904:00 +CPU= 59.527
  06:22:23 STEP  905 H= 905:00 +CPU= 64.721
  06:24:40 STEP  906 H= 906:00 +CPU=110.984
  06:25:24 STEP  907 H= 907:00 +CPU= 42.124
  06:26:04 STEP  908 H= 908:00 +CPU= 38.550
  06:27:17 STEP  909 H= 909:00 +CPU= 62.497
  06:28:11 STEP  910 H= 910:00 +CPU= 39.006
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:32:29 STEP  911 H= 911:00 +CPU= 36.638
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0588
(argv4) batchid: 984
(argv5) wuid: 12201232
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12201232
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ce4620df70d498f4e7b7b00d853d0eb7 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0588_2015050100_123_984_12201232.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0588_2015050100_123_984_12201232.zip
ic_ancil_file: ic_ancil_12201232
ifsdata_file: ifsdata_12201232
climate_data_file: clim_data_12201232
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_593644e7e0966fbf7e001bbc4bf8f906 to: /var/lib/boinc/slots/2/ic_ancil_12201232.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12201232.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_6de947dd54e088af10d97bf2c689e91a to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12201232.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12201232.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12201232.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12201232.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12201232.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12201232.xml
-- Model is restarting --
Adjusting last_iter, 912, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12201232.xml
last_cpu_time: 48733
upload_file_number: 37
last_iter: 911
last_upload: 3196800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0588_2015050100_123_984_12201232_0_r153834994
The child process has been launched with process id: 109
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220244:1674817364:0.003] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
 ABORT!    1 !! *** WAVE MODEL HAS ABORTED 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3531 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.688] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.688] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.692] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.692] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.695] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.699] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.699] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.699] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          INITMDL 
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.700] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           USERIN 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc/slots/2/oifs_43r3_model.exe' : sigcontextptr=0x7ffc0cc9cea0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with plain hex-dump
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x839273] : 0x839273
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x839879] : 0x839879
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x8c2957] : 0x8c2957
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x81d89e] : 0x81d89e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0x15f8aca] : 0x15f8aca
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0x15d51c8] : 0x15d51c8
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0x166b07f] : 0x166b07f
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0x15d7ff0] : 0x15d7ff0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0x8711e0] : 0x8711e0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x7e0165] : 0x7e0165
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0x55e4ad] : 0x55e4ad
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0x5471c9] : 0x5471c9
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x546412] : 0x546412
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x545e28] : 0x545e28
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x40708d] : 0x40708d
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x40220f] : 0x40220f
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [16]: [0x4021a2] : 0x4021a2
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [17]: [0x1dca390] : 0x1dca390
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [18]: [0x40206e] : 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.765] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3531MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x6d], nsigs = 1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220632:1674817592:227.767] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220633:1674817593:228.767] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:Tower:1:1:109:109] [20230127:220633:1674817593:228.767] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC903B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008C2880  sdl_mod_mp_sdl_sr         100  sdl_mod.F90
oifs_43r3_model.e  000000000081D8AF  abor1_                     41  abor1.F90
oifs_43r3_model.e  00000000015F8ACA  mpabort_                   50  mpabort.F
oifs_43r3_model.e  00000000015D51C8  abort1_                    44  abort1.F
oifs_43r3_model.e  000000000166B07F  userin_                   553  userin.F
oifs_43r3_model.e  00000000015D7FF0  initmdl_                  452  initmdl.F
oifs_43r3_model.e  00000000008711E0  wavemdl_                  393  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:41:01 1513123 22287519 oifs_43r3_ps_0588_2015050100_123_984_12201232_0 2,937,600 43,502 0.0148
09 May 2023 16:39:44 1513123 22287519 oifs_43r3_ps_0588_2015050100_123_984_12201232_0 1,728,000 23,062 0.0133
09 May 2023 16:39:31 1513123 22287519 oifs_43r3_ps_0588_2015050100_123_984_12201232_0 864,000 11,425 0.0132


©2024 climateprediction.net