climateprediction.net home page
Task 22249295

Task 22249295

Name oifs_43r3_ps_0300_2021050100_123_945_12163389_2
Workunit 12163389
Created 5 Dec 2022, 3:37:36 UTC
Sent 5 Dec 2022, 3:41:02 UTC
Report deadline 4 Jan 2023, 3:41:02 UTC
Received 6 Dec 2022, 16:07:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1459268
Run time 1 days 11 hours 12 min 56 sec
CPU time 23 min 42 sec
Validate state Invalid
Credit 9,918.72
Device peak FLOPS 2.62 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,406.60 MB
Peak swap size 4,977.92 MB
Peak disk usage 1,214.04 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
c-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_73.zip
Uploading the intermediate file: upload_file_73.zip
Uploading trickle at timestep: 6393600
  04:47:18 STEP 1777 H=1777:00 +CPU= 53.118
  04:48:11 STEP 1778 H=1778:00 +CPU= 53.064
  04:49:32 STEP 1779 H=1779:00 +CPU= 81.332
  04:50:26 STEP 1780 H=1780:00 +CPU= 53.088
  04:51:16 STEP 1781 H=1781:00 +CPU= 50.127
  04:52:39 STEP 1782 H=1782:00 +CPU= 82.903
  04:53:31 STEP 1783 H=1783:00 +CPU= 51.745
  04:54:25 STEP 1784 H=1784:00 +CPU= 53.764
  04:55:47 STEP 1785 H=1785:00 +CPU= 81.877
  04:56:42 STEP 1786 H=1786:00 +CPU= 54.426
  04:57:34 STEP 1787 H=1787:00 +CPU= 52.083
  04:59:05 STEP 1788 H=1788:00 +CPU= 90.063
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001788
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001788
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001788
  04:59:57 STEP 1789 H=1789:00 +CPU= 51.575
  05:00:51 STEP 1790 H=1790:00 +CPU= 53.745
  05:02:13 STEP 1791 H=1791:00 +CPU= 81.954
  05:03:07 STEP 1792 H=1792:00 +CPU= 53.763
  05:04:01 STEP 1793 H=1793:00 +CPU= 53.121
  05:05:25 STEP 1794 H=1794:00 +CPU= 83.890
  05:06:18 STEP 1795 H=1795:00 +CPU= 52.763
  05:07:13 STEP 1796 H=1796:00 +CPU= 54.576
  05:08:36 STEP 1797 H=1797:00 +CPU= 82.541
  05:09:30 STEP 1798 H=1798:00 +CPU= 53.318
  05:10:24 STEP 1799 H=1799:00 +CPU= 54.581
  05:12:00 STEP 1800 H=1800:00 +CPU= 95.199
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001800
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001800
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_74.zip
Uploading the intermediate file: upload_file_74.zip
Uploading trickle at timestep: 6480000
  05:12:53 STEP 1801 H=1801:00 +CPU= 52.695
  05:13:47 STEP 1802 H=1802:00 +CPU= 53.865
  05:15:08 STEP 1803 H=1803:00 +CPU= 80.367
  05:16:01 STEP 1804 H=1804:00 +CPU= 52.543
  05:16:52 STEP 1805 H=1805:00 +CPU= 51.250
  05:18:16 STEP 1806 H=1806:00 +CPU= 82.689
  05:19:08 STEP 1807 H=1807:00 +CPU= 52.240
  05:20:02 STEP 1808 H=1808:00 +CPU= 53.511
  05:21:25 STEP 1809 H=1809:00 +CPU= 82.338
  05:22:16 STEP 1810 H=1810:00 +CPU= 51.774
  05:23:06 STEP 1811 H=1811:00 +CPU= 49.453
  05:24:37 STEP 1812 H=1812:00 +CPU= 90.164
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001812
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001812
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001812
  05:25:30 STEP 1813 H=1813:00 +CPU= 53.408
  05:26:24 STEP 1814 H=1814:00 +CPU= 53.380
  05:27:45 STEP 1815 H=1815:00 +CPU= 80.832
  05:28:39 STEP 1816 H=1816:00 +CPU= 53.437
  05:29:31 STEP 1817 H=1817:00 +CPU= 51.342
  05:30:54 STEP 1818 H=1818:00 +CPU= 82.964
  05:31:46 STEP 1819 H=1819:00 +CPU= 52.006
  05:32:40 STEP 1820 H=1820:00 +CPU= 53.723
  05:34:02 STEP 1821 H=1821:00 +CPU= 81.637
  05:34:57 STEP 1822 H=1822:00 +CPU= 54.391
  05:35:51 STEP 1823 H=1823:00 +CPU= 53.660
  05:37:26 STEP 1824 H=1824:00 +CPU= 94.957
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001824
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001824
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_75.zip
Uploading the intermediate file: upload_file_75.zip
Uploading trickle at timestep: 6566400
  05:38:19 STEP 1825 H=1825:00 +CPU= 52.398
  05:39:13 STEP 1826 H=1826:00 +CPU= 54.409
  05:40:34 STEP 1827 H=1827:00 +CPU= 80.672
  05:41:28 STEP 1828 H=1828:00 +CPU= 53.661
  05:42:18 STEP 1829 H=1829:00 +CPU= 50.212
  05:43:40 STEP 1830 H=1830:00 +CPU= 81.328
  05:44:32 STEP 1831 H=1831:00 +CPU= 51.999
  05:45:26 STEP 1832 H=1832:00 +CPU= 52.960
  05:46:48 STEP 1833 H=1833:00 +CPU= 81.715
  05:47:41 STEP 1834 H=1834:00 +CPU= 53.463
  05:48:35 STEP 1835 H=1835:00 +CPU= 53.344
  05:50:07 STEP 1836 H=1836:00 +CPU= 91.763
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001836
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001836
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001836
  05:51:00 STEP 1837 H=1837:00 +CPU= 53.102
  05:51:55 STEP 1838 H=1838:00 +CPU= 54.691
  05:53:18 STEP 1839 H=1839:00 +CPU= 82.422
  05:54:12 STEP 1840 H=1840:00 +CPU= 53.470
  05:55:03 STEP 1841 H=1841:00 +CPU= 51.283
  05:56:27 STEP 1842 H=1842:00 +CPU= 83.198
  05:57:19 STEP 1843 H=1843:00 +CPU= 52.262
  05:58:12 STEP 1844 H=1844:00 +CPU= 52.887
  05:59:35 STEP 1845 H=1845:00 +CPU= 82.614
  06:00:29 STEP 1846 H=1846:00 +CPU= 53.774
  06:01:24 STEP 1847 H=1847:00 +CPU= 54.473
  06:03:02 STEP 1848 H=1848:00 +CPU= 97.719
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001848
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001848
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_76.zip
Uploading the intermediate file: upload_file_76.zip
Uploading trickle at timestep: 6652800
  06:03:56 STEP 1849 H=1849:00 +CPU= 53.520
  06:04:50 STEP 1850 H=1850:00 +CPU= 54.082
  06:06:13 STEP 1851 H=1851:00 +CPU= 82.490
  06:07:07 STEP 1852 H=1852:00 +CPU= 53.387
  06:08:00 STEP 1853 H=1853:00 +CPU= 52.916
  06:09:25 STEP 1854 H=1854:00 +CPU= 84.387
  06:10:18 STEP 1855 H=1855:00 +CPU= 53.155
  06:11:13 STEP 1856 H=1856:00 +CPU= 54.592
  06:12:37 STEP 1857 H=1857:00 +CPU= 83.239
  06:13:32 STEP 1858 H=1858:00 +CPU= 54.808
  06:14:26 STEP 1859 H=1859:00 +CPU= 53.782
  06:15:56 STEP 1860 H=1860:00 +CPU= 90.065
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001860
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001860
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001860
  06:16:50 STEP 1861 H=1861:00 +CPU= 53.398
  06:17:44 STEP 1862 H=1862:00 +CPU= 53.831
  06:19:07 STEP 1863 H=1863:00 +CPU= 82.370
  06:20:02 STEP 1864 H=1864:00 +CPU= 54.621
  06:20:55 STEP 1865 H=1865:00 +CPU= 52.922
  06:22:19 STEP 1866 H=1866:00 +CPU= 83.037
  06:23:12 STEP 1867 H=1867:00 +CPU= 53.364
  06:24:07 STEP 1868 H=1868:00 +CPU= 54.562
  06:25:31 STEP 1869 H=1869:00 +CPU= 83.005
  06:26:25 STEP 1870 H=1870:00 +CPU= 54.242
  06:27:20 STEP 1871 H=1871:00 +CPU= 54.299
  06:28:56 STEP 1872 H=1872:00 +CPU= 95.783
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001872
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001872
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_77.zip
Uploading the intermediate file: upload_file_77.zip
Uploading trickle at timestep: 6739200
  06:29:49 STEP 1873 H=1873:00 +CPU= 52.895
  06:30:44 STEP 1874 H=1874:00 +CPU= 54.095
  06:32:07 STEP 1875 H=1875:00 +CPU= 83.268
  06:33:02 STEP 1876 H=1876:00 +CPU= 54.268
  06:33:56 STEP 1877 H=1877:00 +CPU= 53.319
  06:35:20 STEP 1878 H=1878:00 +CPU= 83.563
  06:36:13 STEP 1879 H=1879:00 +CPU= 52.818
  06:37:07 STEP 1880 H=1880:00 +CPU= 53.910
  06:38:30 STEP 1881 H=1881:00 +CPU= 82.736
  06:39:25 STEP 1882 H=1882:00 +CPU= 54.822
  06:40:20 STEP 1883 H=1883:00 +CPU= 53.911
  06:41:51 STEP 1884 H=1884:00 +CPU= 90.821
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001884
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001884
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001884
  06:42:44 STEP 1885 H=1885:00 +CPU= 53.056
  06:43:40 STEP 1886 H=1886:00 +CPU= 54.910
  06:45:03 STEP 1887 H=1887:00 +CPU= 83.372
  06:45:58 STEP 1888 H=1888:00 +CPU= 53.981
  06:46:51 STEP 1889 H=1889:00 +CPU= 53.067
  06:48:15 STEP 1890 H=1890:00 +CPU= 83.337
  06:49:08 STEP 1891 H=1891:00 +CPU= 52.850
  06:50:03 STEP 1892 H=1892:00 +CPU= 54.556
  06:51:26 STEP 1893 H=1893:00 +CPU= 82.423
  06:52:21 STEP 1894 H=1894:00 +CPU= 54.457
  06:53:16 STEP 1895 H=1895:00 +CPU= 54.722
  06:54:53 STEP 1896 H=1896:00 +CPU= 96.914
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001896
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001896
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_78.zip
Uploading the intermediate file: upload_file_78.zip
Uploading trickle at timestep: 6825600
  06:55:46 STEP 1897 H=1897:00 +CPU= 52.433
  06:56:40 STEP 1898 H=1898:00 +CPU= 54.217
  06:58:03 STEP 1899 H=1899:00 +CPU= 82.629
  06:58:57 STEP 1900 H=1900:00 +CPU= 53.320
  06:59:50 STEP 1901 H=1901:00 +CPU= 52.830
  07:01:13 STEP 1902 H=1902:00 +CPU= 82.532
  07:02:06 STEP 1903 H=1903:00 +CPU= 53.310
  07:03:02 STEP 1904 H=1904:00 +CPU= 55.014
  07:04:26 STEP 1905 H=1905:00 +CPU= 83.396
  07:05:20 STEP 1906 H=1906:00 +CPU= 54.051
  07:06:14 STEP 1907 H=1907:00 +CPU= 53.262
  07:07:46 STEP 1908 H=1908:00 +CPU= 91.864
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001908
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001908
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001908
  07:08:40 STEP 1909 H=1909:00 +CPU= 53.434
  07:09:35 STEP 1910 H=1910:00 +CPU= 54.465
  07:10:58 STEP 1911 H=1911:00 +CPU= 82.509
  07:11:53 STEP 1912 H=1912:00 +CPU= 54.655
  07:12:46 STEP 1913 H=1913:00 +CPU= 52.899
  07:14:11 STEP 1914 H=1914:00 +CPU= 84.301
  07:15:04 STEP 1915 H=1915:00 +CPU= 52.829
  07:15:58 STEP 1916 H=1916:00 +CPU= 54.271
  07:17:22 STEP 1917 H=1917:00 +CPU= 82.799
  07:18:17 STEP 1918 H=1918:00 +CPU= 54.813
  07:19:12 STEP 1919 H=1919:00 +CPU= 55.295
  07:20:51 STEP 1920 H=1920:00 +CPU= 98.078
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001920
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001920
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_79.zip
Uploading the intermediate file: upload_file_79.zip
Uploading trickle at timestep: 6912000
  07:21:46 STEP 1921 H=1921:00 +CPU= 54.027
  07:22:40 STEP 1922 H=1922:00 +CPU= 54.542
  07:24:04 STEP 1923 H=1923:00 +CPU= 83.372
  07:24:59 STEP 1924 H=1924:00 +CPU= 54.965
  07:25:53 STEP 1925 H=1925:00 +CPU= 53.355
  07:27:18 STEP 1926 H=1926:00 +CPU= 84.603
  07:28:12 STEP 1927 H=1927:00 +CPU= 53.332
  07:29:07 STEP 1928 H=1928:00 +CPU= 55.073
  07:30:31 STEP 1929 H=1929:00 +CPU= 83.283
  07:31:27 STEP 1930 H=1930:00 +CPU= 55.072
  07:32:20 STEP 1931 H=1931:00 +CPU= 53.169
  07:33:52 STEP 1932 H=1932:00 +CPU= 91.785
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001932
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001932
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001932
  07:34:46 STEP 1933 H=1933:00 +CPU= 53.278
  07:35:42 STEP 1934 H=1934:00 +CPU= 55.400
  07:37:05 STEP 1935 H=1935:00 +CPU= 82.754
  07:38:00 STEP 1936 H=1936:00 +CPU= 54.780
  07:38:53 STEP 1937 H=1937:00 +CPU= 52.890
  07:40:18 STEP 1938 H=1938:00 +CPU= 84.414
  07:41:11 STEP 1939 H=1939:00 +CPU= 52.901
  07:42:06 STEP 1940 H=1940:00 +CPU= 54.279
  07:43:27 STEP 1941 H=1941:00 +CPU= 80.913
  07:44:22 STEP 1942 H=1942:00 +CPU= 54.603
  07:45:15 STEP 1943 H=1943:00 +CPU= 53.626
  07:46:50 STEP 1944 H=1944:00 +CPU= 94.219
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001944
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001944
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_80.zip
Uploading the intermediate file: upload_file_80.zip
Uploading trickle at timestep: 6998400
  07:47:44 STEP 1945 H=1945:00 +CPU= 53.217
  07:48:38 STEP 1946 H=1946:00 +CPU= 53.834
  07:50:00 STEP 1947 H=1947:00 +CPU= 81.777
  07:50:55 STEP 1948 H=1948:00 +CPU= 54.895
  07:51:48 STEP 1949 H=1949:00 +CPU= 53.170
  07:53:12 STEP 1950 H=1950:00 +CPU= 83.111
  07:54:06 STEP 1951 H=1951:00 +CPU= 54.135
  07:55:01 STEP 1952 H=1952:00 +CPU= 54.323
  07:56:22 STEP 1953 H=1953:00 +CPU= 81.489
  07:57:15 STEP 1954 H=1954:00 +CPU= 52.703
  07:58:08 STEP 1955 H=1955:00 +CPU= 52.180
  07:59:39 STEP 1956 H=1956:00 +CPU= 90.815
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001956
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001956
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001956
  08:00:32 STEP 1957 H=1957:00 +CPU= 52.876
  08:01:27 STEP 1958 H=1958:00 +CPU= 54.221
  08:02:49 STEP 1959 H=1959:00 +CPU= 82.163
  08:03:44 STEP 1960 H=1960:00 +CPU= 54.579
  08:04:37 STEP 1961 H=1961:00 +CPU= 52.450
  08:06:01 STEP 1962 H=1962:00 +CPU= 83.670
  08:06:54 STEP 1963 H=1963:00 +CPU= 52.741
  08:07:48 STEP 1964 H=1964:00 +CPU= 53.733
  08:09:11 STEP 1965 H=1965:00 +CPU= 82.541
  08:10:07 STEP 1966 H=1966:00 +CPU= 55.100
  08:11:03 STEP 1967 H=1967:00 +CPU= 55.473
  08:12:41 STEP 1968 H=1968:00 +CPU= 98.017
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001968
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001968
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMGGhpi1+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMSHhpi1+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389/ICMUAhpi1+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013_81.zip
Uploading the intermediate file: upload_file_81.zip
Uploading trickle at timestep: 7084800
  08:13:35 STEP 1969 H=1969:00 +CPU= 53.487
  08:14:30 STEP 1970 H=1970:00 +CPU= 54.144
  08:15:53 STEP 1971 H=1971:00 +CPU= 82.801
  08:16:48 STEP 1972 H=1972:00 +CPU= 54.850
  08:17:42 STEP 1973 H=1973:00 +CPU= 53.332
  08:19:06 STEP 1974 H=1974:00 +CPU= 84.184
  08:20:00 STEP 1975 H=1975:00 +CPU= 53.435
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.832] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4420MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.832] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.832] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.832] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.832] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.832] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4420 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082020:1670336420:124729.833] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082021:1670336421:124730.833] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:40574:40574] [20221206:082021:1670336421:124730.833] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
master.exe         0000000001CE916B  Unknown               Unknown  Unknown
master.exe         000000000081FFF1  Unknown               Unknown  Unknown
master.exe         0000000001DC9090  Unknown               Unknown  Unknown
master.exe         0000000000F28A93  callpar_                 1674  callpar.F90
master.exe         0000000000E4940F  ec_phys_                  670  ec_phys.F90
master.exe         0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
master.exe         0000000000BFA3D3  gp_model_                 613  gp_model.F90
master.exe         00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
master.exe         0000000000BBC1CE  scan2m_                   535  scan2m.F90
master.exe         000000000057A664  stepo_                    327  stepo.F90
master.exe         000000000055DEEC  cnt4_                    1133  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0300
(argv4) batchid: 945
(argv5) wuid: 12163389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e9bb1a666703c9b951933845192e35a6 to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
ic_ancil_file: ic_ancil_12163389
ifsdata_file: ifsdata_12163389
climate_data_file: clim_data_12163389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_841db2bdff2fd930f18350aea1dc8f41 to: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_26827761c8a1cf611061fdf79791063d to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12163389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12163389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12163389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12163389.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013
The child process has been launched with process id: 45520
Executing the command: /var/lib/boinc-client/slots/23/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:084035:1670337635:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:42:12 STEP 1968 H=1968:00 +CPU= 91.028
  08:43:05 STEP 1969 H=1969:00 +CPU= 52.948
  08:44:00 STEP 1970 H=1970:00 +CPU= 54.226
  08:45:22 STEP 1971 H=1971:00 +CPU= 81.776
  08:46:16 STEP 1972 H=1972:00 +CPU= 54.207
  08:47:09 STEP 1973 H=1973:00 +CPU= 52.878
  08:48:33 STEP 1974 H=1974:00 +CPU= 83.069
  08:49:25 STEP 1975 H=1975:00 +CPU= 52.130
  08:50:19 STEP 1976 H=1976:00 +CPU= 53.409
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4412MB (heap), 4394MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 4394 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.924] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
double free or corruption (top)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085029:1670338229:593.925] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-INPUT LOOP2  (1206) 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085030:1670338230:594.925] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45520:45520] [20221206:085030:1670338230:594.925] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
master.exe         0000000001CE916B  Unknown               Unknown  Unknown
master.exe         000000000081FFF1  Unknown               Unknown  Unknown
master.exe         0000000001DC9090  Unknown               Unknown  Unknown
master.exe         00000000012BD275  radintg_                  945  radintg.F90
master.exe         0000000001270124  raddrv_                   537  raddrv.F90
master.exe         0000000000BFD839  gp_model_                 482  gp_model.F90
master.exe         00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
master.exe         0000000000BBC1CE  scan2m_                   535  scan2m.F90
master.exe         000000000057A664  stepo_                    327  stepo.F90
master.exe         000000000055DEEC  cnt4_                    1133  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0300
(argv4) batchid: 945
(argv5) wuid: 12163389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e9bb1a666703c9b951933845192e35a6 to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
ic_ancil_file: ic_ancil_12163389
ifsdata_file: ifsdata_12163389
climate_data_file: clim_data_12163389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_841db2bdff2fd930f18350aea1dc8f41 to: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_26827761c8a1cf611061fdf79791063d to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12163389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12163389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12163389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12163389.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0300_2021050100_123_945_12163389_2_r786179013
The child process has been launched with process id: 45721
Executing the command: /var/lib/boinc-client/slots/23/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:091115:1670339475:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:12:55 STEP 1968 H=1968:00 +CPU= 93.665
  09:13:48 STEP 1969 H=1969:00 +CPU= 52.737
  09:14:44 STEP 1970 H=1970:00 +CPU= 54.722
  09:16:07 STEP 1971 H=1971:00 +CPU= 82.816
  09:17:02 STEP 1972 H=1972:00 +CPU= 54.727
  09:17:55 STEP 1973 H=1973:00 +CPU= 53.083
  09:19:19 STEP 1974 H=1974:00 +CPU= 82.964
  09:20:11 STEP 1975 H=1975:00 +CPU= 51.968
  09:21:05 STEP 1976 H=1976:00 +CPU= 53.757
  09:22:28 STEP 1977 H=1977:00 +CPU= 82.238
  09:23:23 STEP 1978 H=1978:00 +CPU= 54.300
  09:24:17 STEP 1979 H=1979:00 +CPU= 53.891
  09:25:49 STEP 1980 H=1980:00 +CPU= 91.030
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGhpi1+001980
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHhpi1+001980
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAhpi1+001980
  09:26:42 STEP 1981 H=1981:00 +CPU= 53.027
  09:27:36 STEP 1982 H=1982:00 +CPU= 54.054
  09:28:58 STEP 1983 H=1983:00 +CPU= 81.390
  09:29:53 STEP 1984 H=1984:00 +CPU= 54.701
  09:30:47 STEP 1985 H=1985:00 +CPU= 53.001
  09:32:11 STEP 1986 H=1986:00 +CPU= 83.408
  09:33:04 STEP 1987 H=1987:00 +CPU= 52.835
  09:33:58 STEP 1988 H=1988:00 +CPU= 53.641
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4411MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4411 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                SURFTSTP_LAYER 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 SURFTSTP 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  SURFTSTP_CTL_MOD:SURFTSTP_CTL 
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093508:1670340908:1432.974] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                   SRFWEXC_VG_MOD:SRFWEXC_VG 
double free or corruption (top)
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093509:1670340909:1433.975] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:boinc-00.fullford.com:1:1:45721:45721] [20221206:093509:1670340909:1433.975] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
master.exe         0000000001CE916B  Unknown               Unknown  Unknown
master.exe         000000000081FFF1  Unknown               Unknown  Unknown
master.exe         0000000001DC9090  Unknown               Unknown  Unknown
master.exe         0000000001CC3653  Unknown               Unknown  Unknown
master.exe         000000000176D4E0  srfwexc_vg_mod_mp         314  srfwexc_vg_mod.F90
master.exe         000000000174461F  surftstp_ctl_mod_         699  surftstp_ctl_mod.F90
master.exe         00000000015CB81F  surftstp_                1018  surftstp.F90
master.exe         00000000011E89C5  surftstp_layer_           148  surftstp_layer.F90
master.exe         0000000000F25139  callpar_                 1516  callpar.F90
master.exe         0000000000E4940F  ec_phys_                  670  ec_phys.F90
master.exe         0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
master.exe         0000000000BFA3D3  gp_model_                 613  gp_model.F90
master.exe         00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
master.exe         0000000000BBC1CE  scan2m_                   535  scan2m.F90
master.exe         000000000057A664  stepo_                    327  stepo.F90
master.exe         000000000055DEEC  cnt4_                    1133  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0300
(argv4) batchid: 945
(argv5) wuid: 12163389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e9bb1a666703c9b951933845192e35a6 to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
ic_ancil_file: ic_ancil_12163389
ifsdata_file: ifsdata_12163389
climate_data_file: clim_data_12163389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_841db2bdff2fd930f18350aea1dc8f41 to: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0300
(argv4) batchid: 945
(argv5) wuid: 12163389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e9bb1a666703c9b951933845192e35a6 to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0300_2021050100_123_945_12163389.zip
ic_ancil_file: ic_ancil_12163389
ifsdata_file: ifsdata_12163389
climate_data_file: clim_data_12163389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from:  #&#212;&#199;h[&#197;wK^&#225;&#187;&#133;&#249;&#218;&#242;&#255;&#215;*&#249;&#132;+&#215;&#174;&#234;7&#242;?&#197;&#202;&#191;&#243;&#255;&#254;&#253;?&#140;&#140;&#255;&#253;&#254;&#223;&#245;_&#215;&#232;&#211;4&#181;&#218;&#149;g&#187;&#208;n:Ht0WUl&#142;&#152;&#255;&#167;9&#243;&#175;&#230;&#249;&#250;&#161;&#230;&#226;&#233;&#230;&#167;&#168;=&#214;&#224;&#217;&,|&#194;&#156;&#207;	l&#202;&#173;=&#216;&#198;Y!&#216;&#165;&#166;c&#240;&#202;&#157;&#241;&#142;&#186;&#156; to: /var/lib/boinc-client/slots/23/ic_ancil_12163389.zip
..Copying the IC ancils to the working directory failed

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:35:39 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 7,084,800 123,490 0.0174
09 May 2023 15:35:38 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,998,400 121,953 0.0174
09 May 2023 15:35:38 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,912,000 120,396 0.0174
09 May 2023 15:35:36 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,825,600 118,850 0.0174
09 May 2023 15:35:36 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,739,200 117,304 0.0174
09 May 2023 15:35:36 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,652,800 115,758 0.0174
09 May 2023 15:35:33 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,566,400 114,229 0.0174
09 May 2023 15:35:33 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,480,000 112,712 0.0174
09 May 2023 15:35:31 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,393,600 111,174 0.0174
09 May 2023 15:35:31 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,307,200 109,658 0.0174
09 May 2023 15:35:29 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,220,800 108,132 0.0174
09 May 2023 15:35:29 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,134,400 106,633 0.0174
09 May 2023 15:35:29 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 6,048,000 105,109 0.0174
09 May 2023 15:35:27 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,961,600 103,569 0.0174
09 May 2023 15:35:27 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,875,200 102,021 0.0174
09 May 2023 15:35:25 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,788,800 100,510 0.0174
09 May 2023 15:35:25 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,702,400 99,022 0.0174
09 May 2023 15:35:22 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,616,000 97,482 0.0174
09 May 2023 15:35:22 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,529,600 95,972 0.0174
09 May 2023 15:35:22 1459268 22249295 oifs_43r3_ps_0300_2021050100_123_945_12163389_2 5,443,200 94,425 0.0173


©2024 climateprediction.net