climateprediction.net home page
Task 22245657

Task 22245657

Name oifs_43r3_ps_0624_2021050100_123_945_12163713_0
Workunit 12163713
Created 28 Nov 2022, 20:47:19 UTC
Sent 29 Nov 2022, 0:45:42 UTC
Report deadline 29 Dec 2022, 0:45:42 UTC
Received 29 Nov 2022, 22:36:40 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1378058
Run time 16 hours 56 min 26 sec
CPU time 16 hours 34 min 37 sec
Validate state Invalid
Credit 11,128.32
Device peak FLOPS 4.96 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,415.23 MB
Peak swap size 4,974.26 MB
Peak disk usage 1,221.05 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ts/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001764
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001764
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_73.zip
Uploading the intermediate file: upload_file_73.zip
  11:59:51 STEP 1777 H=1777:00 +CPU= 18.784
Uploading trickle at timestep: 6393600
  12:00:12 STEP 1778 H=1778:00 +CPU= 20.791
  12:00:45 STEP 1779 H=1779:00 +CPU= 33.184
  12:01:07 STEP 1780 H=1780:00 +CPU= 21.226
  12:01:29 STEP 1781 H=1781:00 +CPU= 22.553
  12:02:03 STEP 1782 H=1782:00 +CPU= 33.413
  12:02:25 STEP 1783 H=1783:00 +CPU= 22.165
  12:02:47 STEP 1784 H=1784:00 +CPU= 22.430
  12:03:21 STEP 1785 H=1785:00 +CPU= 33.684
  12:03:42 STEP 1786 H=1786:00 +CPU= 21.073
  12:04:04 STEP 1787 H=1787:00 +CPU= 21.647
  12:04:43 STEP 1788 H=1788:00 +CPU= 38.656
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001788
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001788
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001788
  12:05:04 STEP 1789 H=1789:00 +CPU= 21.056
  12:05:26 STEP 1790 H=1790:00 +CPU= 21.979
  12:05:59 STEP 1791 H=1791:00 +CPU= 32.712
  12:06:19 STEP 1792 H=1792:00 +CPU= 20.448
  12:06:41 STEP 1793 H=1793:00 +CPU= 22.206
  12:07:14 STEP 1794 H=1794:00 +CPU= 32.517
  12:07:29 STEP 1795 H=1795:00 +CPU= 15.628
  12:07:51 STEP 1796 H=1796:00 +CPU= 21.595
  12:08:25 STEP 1797 H=1797:00 +CPU= 34.350
  12:08:48 STEP 1798 H=1798:00 +CPU= 22.832
  12:09:11 STEP 1799 H=1799:00 +CPU= 22.486
  12:09:50 STEP 1800 H=1800:00 +CPU= 39.253
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001800
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001800
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001800
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001776
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001776
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001776
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001788
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001788
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001788
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_74.zip
Uploading the intermediate file: upload_file_74.zip
  12:10:10 STEP 1801 H=1801:00 +CPU= 19.511
Uploading trickle at timestep: 6480000
  12:10:29 STEP 1802 H=1802:00 +CPU= 19.252
  12:11:01 STEP 1803 H=1803:00 +CPU= 32.262
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:22 STEP 1804 H=1804:00 +CPU= 18.484
  12:18:41 STEP 1805 H=1805:00 +CPU= 19.172
  12:19:10 STEP 1806 H=1806:00 +CPU= 28.273
  12:19:26 STEP 1807 H=1807:00 +CPU= 15.933
  12:19:45 STEP 1808 H=1808:00 +CPU= 19.609
  12:20:13 STEP 1809 H=1809:00 +CPU= 27.419
  12:20:29 STEP 1810 H=1810:00 +CPU= 16.498
  12:20:46 STEP 1811 H=1811:00 +CPU= 16.798
  12:21:18 STEP 1812 H=1812:00 +CPU= 31.919
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001812
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001812
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001812
  12:21:34 STEP 1813 H=1813:00 +CPU= 16.333
  12:21:52 STEP 1814 H=1814:00 +CPU= 18.071
  12:22:19 STEP 1815 H=1815:00 +CPU= 26.330
  12:22:35 STEP 1816 H=1816:00 +CPU= 16.461
  12:22:51 STEP 1817 H=1817:00 +CPU= 15.970
  12:23:17 STEP 1818 H=1818:00 +CPU= 25.340
  12:23:34 STEP 1819 H=1819:00 +CPU= 17.129
  12:23:51 STEP 1820 H=1820:00 +CPU= 17.108
  12:24:18 STEP 1821 H=1821:00 +CPU= 26.906
  12:24:36 STEP 1822 H=1822:00 +CPU= 17.772
  12:24:54 STEP 1823 H=1823:00 +CPU= 18.029
  12:25:25 STEP 1824 H=1824:00 +CPU= 30.952
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001824
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001824
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001824
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001800
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001800
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001800
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001812
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001812
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001812
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_75.zip
Uploading the intermediate file: upload_file_75.zip
  12:25:41 STEP 1825 H=1825:00 +CPU= 15.824
Uploading trickle at timestep: 6566400
  12:25:58 STEP 1826 H=1826:00 +CPU= 17.165
  12:26:25 STEP 1827 H=1827:00 +CPU= 27.044
  12:26:43 STEP 1828 H=1828:00 +CPU= 18.546
  12:27:01 STEP 1829 H=1829:00 +CPU= 17.881
  12:27:27 STEP 1830 H=1830:00 +CPU= 26.098
  12:27:43 STEP 1831 H=1831:00 +CPU= 15.831
  12:28:00 STEP 1832 H=1832:00 +CPU= 16.730
  12:28:31 STEP 1833 H=1833:00 +CPU= 31.391
  12:28:55 STEP 1834 H=1834:00 +CPU= 23.401
  12:29:16 STEP 1835 H=1835:00 +CPU= 21.461
  12:29:54 STEP 1836 H=1836:00 +CPU= 36.970
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001836
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001836
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001836
  12:30:14 STEP 1837 H=1837:00 +CPU= 19.971
  12:30:36 STEP 1838 H=1838:00 +CPU= 21.348
  12:31:07 STEP 1839 H=1839:00 +CPU= 31.357
  12:31:29 STEP 1840 H=1840:00 +CPU= 22.405
  12:31:50 STEP 1841 H=1841:00 +CPU= 20.833
  12:32:25 STEP 1842 H=1842:00 +CPU= 34.841
  12:32:47 STEP 1843 H=1843:00 +CPU= 21.313
  12:33:09 STEP 1844 H=1844:00 +CPU= 22.205
  12:33:42 STEP 1845 H=1845:00 +CPU= 33.318
  12:34:06 STEP 1846 H=1846:00 +CPU= 23.428
  12:34:29 STEP 1847 H=1847:00 +CPU= 22.867
  12:35:05 STEP 1848 H=1848:00 +CPU= 36.316
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001848
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001848
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001848
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001824
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001824
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001824
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001836
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001836
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001836
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_76.zip
Uploading the intermediate file: upload_file_76.zip
  12:35:26 STEP 1849 H=1849:00 +CPU= 20.984
Uploading trickle at timestep: 6652800
  12:35:48 STEP 1850 H=1850:00 +CPU= 21.626
  12:36:28 STEP 1851 H=1851:00 +CPU= 39.765
  12:36:53 STEP 1852 H=1852:00 +CPU= 25.215
  12:37:24 STEP 1853 H=1853:00 +CPU= 30.539
  12:38:01 STEP 1854 H=1854:00 +CPU= 37.401
  12:38:31 STEP 1855 H=1855:00 +CPU= 29.415
  12:39:00 STEP 1856 H=1856:00 +CPU= 28.552
  12:39:33 STEP 1857 H=1857:00 +CPU= 33.844
  12:40:05 STEP 1858 H=1858:00 +CPU= 31.800
  12:40:27 STEP 1859 H=1859:00 +CPU= 21.884
  12:41:20 STEP 1860 H=1860:00 +CPU= 50.737
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001860
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001860
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001860
  12:41:41 STEP 1861 H=1861:00 +CPU= 21.204
  12:42:04 STEP 1862 H=1862:00 +CPU= 22.123
  12:42:38 STEP 1863 H=1863:00 +CPU= 34.767
  12:42:59 STEP 1864 H=1864:00 +CPU= 20.756
  12:43:21 STEP 1865 H=1865:00 +CPU= 21.568
  12:43:56 STEP 1866 H=1866:00 +CPU= 34.623
  12:44:15 STEP 1867 H=1867:00 +CPU= 19.316
  12:44:38 STEP 1868 H=1868:00 +CPU= 23.351
  12:45:12 STEP 1869 H=1869:00 +CPU= 34.189
  12:45:34 STEP 1870 H=1870:00 +CPU= 21.037
  12:45:56 STEP 1871 H=1871:00 +CPU= 22.390
  12:46:35 STEP 1872 H=1872:00 +CPU= 39.199
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001872
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001872
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001872
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001848
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001848
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001848
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001860
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001860
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001860
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_77.zip
Uploading the intermediate file: upload_file_77.zip
  12:46:58 STEP 1873 H=1873:00 +CPU= 22.897
Uploading trickle at timestep: 6739200
  12:47:21 STEP 1874 H=1874:00 +CPU= 22.361
  12:47:54 STEP 1875 H=1875:00 +CPU= 33.337
  12:48:19 STEP 1876 H=1876:00 +CPU= 24.355
  12:48:40 STEP 1877 H=1877:00 +CPU= 21.377
  12:49:15 STEP 1878 H=1878:00 +CPU= 35.000
  12:49:38 STEP 1879 H=1879:00 +CPU= 22.648
  12:50:01 STEP 1880 H=1880:00 +CPU= 23.028
  12:50:35 STEP 1881 H=1881:00 +CPU= 34.530
  12:50:59 STEP 1882 H=1882:00 +CPU= 23.254
  12:51:21 STEP 1883 H=1883:00 +CPU= 22.576
  12:52:02 STEP 1884 H=1884:00 +CPU= 40.244
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001884
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001884
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001884
  12:52:25 STEP 1885 H=1885:00 +CPU= 23.558
  12:52:49 STEP 1886 H=1886:00 +CPU= 24.156
  12:53:24 STEP 1887 H=1887:00 +CPU= 34.737
  12:53:48 STEP 1888 H=1888:00 +CPU= 23.467
  12:54:11 STEP 1889 H=1889:00 +CPU= 22.738
  12:54:49 STEP 1890 H=1890:00 +CPU= 37.935
  12:55:11 STEP 1891 H=1891:00 +CPU= 22.380
  12:55:34 STEP 1892 H=1892:00 +CPU= 22.665
  12:56:10 STEP 1893 H=1893:00 +CPU= 36.588
  12:56:33 STEP 1894 H=1894:00 +CPU= 23.156
  12:56:56 STEP 1895 H=1895:00 +CPU= 22.119
  12:57:34 STEP 1896 H=1896:00 +CPU= 37.796
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001896
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001896
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001896
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001872
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001872
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001872
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001884
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001884
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001884
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_78.zip
Uploading the intermediate file: upload_file_78.zip
  12:57:56 STEP 1897 H=1897:00 +CPU= 22.062
Uploading trickle at timestep: 6825600
  12:58:19 STEP 1898 H=1898:00 +CPU= 23.463
  12:58:56 STEP 1899 H=1899:00 +CPU= 37.024
  12:59:20 STEP 1900 H=1900:00 +CPU= 23.127
  12:59:41 STEP 1901 H=1901:00 +CPU= 21.037
  13:00:15 STEP 1902 H=1902:00 +CPU= 34.159
  13:00:35 STEP 1903 H=1903:00 +CPU= 20.438
  13:00:57 STEP 1904 H=1904:00 +CPU= 21.315
  13:01:29 STEP 1905 H=1905:00 +CPU= 32.862
  13:01:52 STEP 1906 H=1906:00 +CPU= 22.625
  13:02:15 STEP 1907 H=1907:00 +CPU= 22.608
  13:02:52 STEP 1908 H=1908:00 +CPU= 37.666
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001908
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001908
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001908
  13:03:15 STEP 1909 H=1909:00 +CPU= 22.843
  13:03:39 STEP 1910 H=1910:00 +CPU= 23.609
  13:04:15 STEP 1911 H=1911:00 +CPU= 35.474
  13:04:37 STEP 1912 H=1912:00 +CPU= 22.375
  13:05:00 STEP 1913 H=1913:00 +CPU= 22.718
  13:05:36 STEP 1914 H=1914:00 +CPU= 36.148
  13:05:58 STEP 1915 H=1915:00 +CPU= 22.465
  13:06:22 STEP 1916 H=1916:00 +CPU= 23.765
  13:06:57 STEP 1917 H=1917:00 +CPU= 34.727
  13:07:21 STEP 1918 H=1918:00 +CPU= 23.556
  13:07:43 STEP 1919 H=1919:00 +CPU= 22.186
  13:08:22 STEP 1920 H=1920:00 +CPU= 39.313
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001920
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001920
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001920
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001896
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001896
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001896
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001908
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001908
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001908
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_79.zip
Uploading the intermediate file: upload_file_79.zip
  13:08:44 STEP 1921 H=1921:00 +CPU= 20.973
Uploading trickle at timestep: 6912000
  13:09:10 STEP 1922 H=1922:00 +CPU= 25.567
  13:09:45 STEP 1923 H=1923:00 +CPU= 34.761
  13:10:07 STEP 1924 H=1924:00 +CPU= 22.681
  13:10:28 STEP 1925 H=1925:00 +CPU= 20.378
  13:11:04 STEP 1926 H=1926:00 +CPU= 35.726
  13:11:27 STEP 1927 H=1927:00 +CPU= 23.759
  13:11:52 STEP 1928 H=1928:00 +CPU= 24.160
  13:12:25 STEP 1929 H=1929:00 +CPU= 33.699
  13:12:51 STEP 1930 H=1930:00 +CPU= 25.426
  13:13:13 STEP 1931 H=1931:00 +CPU= 22.319
  13:13:51 STEP 1932 H=1932:00 +CPU= 38.188
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001932
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001932
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001932
  13:14:15 STEP 1933 H=1933:00 +CPU= 23.231
  13:14:38 STEP 1934 H=1934:00 +CPU= 23.278
  13:15:13 STEP 1935 H=1935:00 +CPU= 35.320
  13:15:38 STEP 1936 H=1936:00 +CPU= 24.824
  13:16:02 STEP 1937 H=1937:00 +CPU= 23.300
  13:16:38 STEP 1938 H=1938:00 +CPU= 36.084
  13:16:59 STEP 1939 H=1939:00 +CPU= 20.892
  13:17:23 STEP 1940 H=1940:00 +CPU= 23.873
  13:17:59 STEP 1941 H=1941:00 +CPU= 36.446
  13:18:24 STEP 1942 H=1942:00 +CPU= 24.428
  13:18:48 STEP 1943 H=1943:00 +CPU= 24.117
  13:19:30 STEP 1944 H=1944:00 +CPU= 42.316
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001944
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001944
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001944
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001920
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001920
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001920
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001932
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001932
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001932
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_80.zip
Uploading the intermediate file: upload_file_80.zip
  13:19:55 STEP 1945 H=1945:00 +CPU= 24.471
Uploading trickle at timestep: 6998400
  13:20:20 STEP 1946 H=1946:00 +CPU= 25.261
  13:20:54 STEP 1947 H=1947:00 +CPU= 34.130
  13:21:18 STEP 1948 H=1948:00 +CPU= 23.943
  13:21:42 STEP 1949 H=1949:00 +CPU= 23.646
  13:22:18 STEP 1950 H=1950:00 +CPU= 35.604
  13:22:41 STEP 1951 H=1951:00 +CPU= 23.218
  13:23:04 STEP 1952 H=1952:00 +CPU= 22.689
  13:23:39 STEP 1953 H=1953:00 +CPU= 35.579
  13:24:02 STEP 1954 H=1954:00 +CPU= 22.671
  13:24:24 STEP 1955 H=1955:00 +CPU= 21.271
  13:25:01 STEP 1956 H=1956:00 +CPU= 37.871
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001956
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001956
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001956
  13:25:24 STEP 1957 H=1957:00 +CPU= 22.965
  13:25:48 STEP 1958 H=1958:00 +CPU= 23.749
  13:26:28 STEP 1959 H=1959:00 +CPU= 35.407
  13:26:51 STEP 1960 H=1960:00 +CPU= 23.003
  13:27:15 STEP 1961 H=1961:00 +CPU= 23.929
  13:27:51 STEP 1962 H=1962:00 +CPU= 35.229
  13:28:13 STEP 1963 H=1963:00 +CPU= 21.756
  13:28:36 STEP 1964 H=1964:00 +CPU= 23.931
  13:29:13 STEP 1965 H=1965:00 +CPU= 36.709
  13:29:38 STEP 1966 H=1966:00 +CPU= 24.623
  13:30:03 STEP 1967 H=1967:00 +CPU= 24.514
  13:30:45 STEP 1968 H=1968:00 +CPU= 42.635
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001968
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001968
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001968
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001944
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001944
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001944
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001956
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001956
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001956
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_81.zip
Uploading the intermediate file: upload_file_81.zip
  13:31:09 STEP 1969 H=1969:00 +CPU= 23.497
Uploading trickle at timestep: 7084800
  13:31:35 STEP 1970 H=1970:00 +CPU= 25.535
  13:32:12 STEP 1971 H=1971:00 +CPU= 36.616
  13:32:36 STEP 1972 H=1972:00 +CPU= 24.254
  13:32:59 STEP 1973 H=1973:00 +CPU= 23.176
  13:33:35 STEP 1974 H=1974:00 +CPU= 36.128
  13:33:59 STEP 1975 H=1975:00 +CPU= 23.653
  13:34:23 STEP 1976 H=1976:00 +CPU= 23.637
  13:34:56 STEP 1977 H=1977:00 +CPU= 33.629
  13:35:18 STEP 1978 H=1978:00 +CPU= 21.863
  13:35:42 STEP 1979 H=1979:00 +CPU= 24.159
  13:36:22 STEP 1980 H=1980:00 +CPU= 39.284
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001980
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001980
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001980
  13:36:40 STEP 1981 H=1981:00 +CPU= 18.384
  13:37:03 STEP 1982 H=1982:00 +CPU= 22.228
  13:37:36 STEP 1983 H=1983:00 +CPU= 33.080
  13:38:00 STEP 1984 H=1984:00 +CPU= 24.588
  13:38:23 STEP 1985 H=1985:00 +CPU= 22.931
  13:38:59 STEP 1986 H=1986:00 +CPU= 36.171
  13:39:21 STEP 1987 H=1987:00 +CPU= 21.836
  13:39:44 STEP 1988 H=1988:00 +CPU= 22.741
  13:40:19 STEP 1989 H=1989:00 +CPU= 34.841
  13:40:41 STEP 1990 H=1990:00 +CPU= 21.839
  13:41:05 STEP 1991 H=1991:00 +CPU= 23.749
  13:41:42 STEP 1992 H=1992:00 +CPU= 37.219
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+001992
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+001992
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+001992
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001968
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001968
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001968
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001980
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001980
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001980
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_82.zip
Uploading the intermediate file: upload_file_82.zip
  13:42:05 STEP 1993 H=1993:00 +CPU= 20.074
Uploading trickle at timestep: 7171200
  13:42:28 STEP 1994 H=1994:00 +CPU= 22.687
  13:43:03 STEP 1995 H=1995:00 +CPU= 35.137
  13:43:26 STEP 1996 H=1996:00 +CPU= 22.927
  13:43:50 STEP 1997 H=1997:00 +CPU= 23.651
  13:44:28 STEP 1998 H=1998:00 +CPU= 38.408
  13:44:53 STEP 1999 H=1999:00 +CPU= 24.440
  13:45:17 STEP 2000 H=2000:00 +CPU= 24.107
  13:45:51 STEP 2001 H=2001:00 +CPU= 34.115
  13:46:15 STEP 2002 H=2002:00 +CPU= 23.989
  13:46:39 STEP 2003 H=2003:00 +CPU= 23.699
  13:47:19 STEP 2004 H=2004:00 +CPU= 40.289
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002004
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002004
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002004
  13:47:42 STEP 2005 H=2005:00 +CPU= 22.365
  13:48:07 STEP 2006 H=2006:00 +CPU= 24.534
  13:48:44 STEP 2007 H=2007:00 +CPU= 37.588
  13:49:08 STEP 2008 H=2008:00 +CPU= 23.443
  13:49:31 STEP 2009 H=2009:00 +CPU= 22.952
  13:50:08 STEP 2010 H=2010:00 +CPU= 36.763
  13:50:37 STEP 2011 H=2011:00 +CPU= 24.173
  13:51:03 STEP 2012 H=2012:00 +CPU= 23.716
  13:51:57 STEP 2013 H=2013:00 +CPU= 37.320
  13:52:32 STEP 2014 H=2014:00 +CPU= 24.231
  13:53:08 STEP 2015 H=2015:00 +CPU= 24.545
  13:54:00 STEP 2016 H=2016:00 +CPU= 42.322
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002016
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002016
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002016
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+001992
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+001992
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+001992
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002004
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002004
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002004
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_83.zip
Uploading the intermediate file: upload_file_83.zip
Uploading trickle at timestep: 7257600
  13:54:26 STEP 2017 H=2017:00 +CPU= 23.932
  13:54:52 STEP 2018 H=2018:00 +CPU= 24.145
  13:55:32 STEP 2019 H=2019:00 +CPU= 35.989
  13:55:59 STEP 2020 H=2020:00 +CPU= 23.868
  13:56:26 STEP 2021 H=2021:00 +CPU= 23.365
  13:57:07 STEP 2022 H=2022:00 +CPU= 38.188
  13:57:33 STEP 2023 H=2023:00 +CPU= 23.463
  13:58:00 STEP 2024 H=2024:00 +CPU= 24.737
  13:58:40 STEP 2025 H=2025:00 +CPU= 36.466
  13:59:08 STEP 2026 H=2026:00 +CPU= 24.367
  13:59:34 STEP 2027 H=2027:00 +CPU= 23.674
  14:00:19 STEP 2028 H=2028:00 +CPU= 40.372
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002028
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002028
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002028
  14:00:46 STEP 2029 H=2029:00 +CPU= 23.463
  14:01:13 STEP 2030 H=2030:00 +CPU= 24.428
  14:01:53 STEP 2031 H=2031:00 +CPU= 36.121
  14:02:21 STEP 2032 H=2032:00 +CPU= 25.043
  14:02:47 STEP 2033 H=2033:00 +CPU= 24.218
  14:03:29 STEP 2034 H=2034:00 +CPU= 37.076
  14:03:56 STEP 2035 H=2035:00 +CPU= 23.408
  14:04:23 STEP 2036 H=2036:00 +CPU= 24.086
  14:05:04 STEP 2037 H=2037:00 +CPU= 35.748
  14:05:31 STEP 2038 H=2038:00 +CPU= 24.332
  14:05:57 STEP 2039 H=2039:00 +CPU= 23.695
  14:06:46 STEP 2040 H=2040:00 +CPU= 43.619
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002040
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002040
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002040
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002016
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002016
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002016
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002028
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002028
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002028
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_84.zip
Uploading the intermediate file: upload_file_84.zip
  14:07:12 STEP 2041 H=2041:00 +CPU= 23.632
Uploading trickle at timestep: 7344000
  14:07:39 STEP 2042 H=2042:00 +CPU= 24.402
  14:08:20 STEP 2043 H=2043:00 +CPU= 36.375
  14:08:48 STEP 2044 H=2044:00 +CPU= 24.223
  14:09:15 STEP 2045 H=2045:00 +CPU= 24.061
  14:09:58 STEP 2046 H=2046:00 +CPU= 36.974
  14:10:24 STEP 2047 H=2047:00 +CPU= 23.704
  14:10:52 STEP 2048 H=2048:00 +CPU= 24.359
  14:11:33 STEP 2049 H=2049:00 +CPU= 37.157
  14:12:02 STEP 2050 H=2050:00 +CPU= 24.814
  14:12:29 STEP 2051 H=2051:00 +CPU= 23.876
  14:13:14 STEP 2052 H=2052:00 +CPU= 40.764
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002052
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002052
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002052
  14:13:41 STEP 2053 H=2053:00 +CPU= 23.791
  14:14:08 STEP 2054 H=2054:00 +CPU= 24.456
  14:14:50 STEP 2055 H=2055:00 +CPU= 35.601
  14:15:17 STEP 2056 H=2056:00 +CPU= 23.901
  14:15:43 STEP 2057 H=2057:00 +CPU= 23.542
  14:16:24 STEP 2058 H=2058:00 +CPU= 38.287
  14:16:50 STEP 2059 H=2059:00 +CPU= 23.560
  14:17:18 STEP 2060 H=2060:00 +CPU= 24.348
  14:17:59 STEP 2061 H=2061:00 +CPU= 36.447
  14:18:26 STEP 2062 H=2062:00 +CPU= 23.954
  14:18:53 STEP 2063 H=2063:00 +CPU= 23.724
  14:19:41 STEP 2064 H=2064:00 +CPU= 42.574
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002064
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002064
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002064
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002040
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002040
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002040
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002052
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002052
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002052
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_85.zip
Uploading the intermediate file: upload_file_85.zip
  14:20:08 STEP 2065 H=2065:00 +CPU= 23.873
Uploading trickle at timestep: 7430400
  14:20:35 STEP 2066 H=2066:00 +CPU= 24.456
  14:21:15 STEP 2067 H=2067:00 +CPU= 36.685
  14:21:43 STEP 2068 H=2068:00 +CPU= 24.181
  14:22:10 STEP 2069 H=2069:00 +CPU= 23.250
  14:22:51 STEP 2070 H=2070:00 +CPU= 37.331
  14:23:17 STEP 2071 H=2071:00 +CPU= 23.009
  14:23:45 STEP 2072 H=2072:00 +CPU= 24.367
  14:24:26 STEP 2073 H=2073:00 +CPU= 36.567
  14:24:54 STEP 2074 H=2074:00 +CPU= 24.690
  14:25:21 STEP 2075 H=2075:00 +CPU= 23.368
  14:26:07 STEP 2076 H=2076:00 +CPU= 40.614
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002076
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002076
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002076
  14:26:32 STEP 2077 H=2077:00 +CPU= 22.870
  14:26:59 STEP 2078 H=2078:00 +CPU= 23.890
  14:27:38 STEP 2079 H=2079:00 +CPU= 35.960
  14:28:05 STEP 2080 H=2080:00 +CPU= 23.765
  14:28:31 STEP 2081 H=2081:00 +CPU= 23.137
  14:29:13 STEP 2082 H=2082:00 +CPU= 37.187
  14:29:39 STEP 2083 H=2083:00 +CPU= 22.962
  14:30:04 STEP 2084 H=2084:00 +CPU= 23.296
  14:30:45 STEP 2085 H=2085:00 +CPU= 36.145
  14:31:11 STEP 2086 H=2086:00 +CPU= 23.247
  14:31:37 STEP 2087 H=2087:00 +CPU= 23.628
  14:32:24 STEP 2088 H=2088:00 +CPU= 41.899
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002088
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002088
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002088
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002064
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002064
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002064
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002076
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002076
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002076
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_86.zip
Uploading the intermediate file: upload_file_86.zip
Uploading trickle at timestep: 7516800
  14:32:50 STEP 2089 H=2089:00 +CPU= 22.737
  14:33:17 STEP 2090 H=2090:00 +CPU= 23.433
  14:33:56 STEP 2091 H=2091:00 +CPU= 35.392
  14:34:23 STEP 2092 H=2092:00 +CPU= 24.350
  14:34:49 STEP 2093 H=2093:00 +CPU= 23.506
  14:35:31 STEP 2094 H=2094:00 +CPU= 37.155
  14:35:58 STEP 2095 H=2095:00 +CPU= 23.583
  14:36:24 STEP 2096 H=2096:00 +CPU= 23.841
  14:37:05 STEP 2097 H=2097:00 +CPU= 36.256
  14:37:32 STEP 2098 H=2098:00 +CPU= 24.143
  14:37:58 STEP 2099 H=2099:00 +CPU= 23.742
  14:38:44 STEP 2100 H=2100:00 +CPU= 40.117
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002100
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002100
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002100
  14:39:11 STEP 2101 H=2101:00 +CPU= 23.251
  14:39:38 STEP 2102 H=2102:00 +CPU= 23.885
  14:40:19 STEP 2103 H=2103:00 +CPU= 35.454
  14:40:46 STEP 2104 H=2104:00 +CPU= 23.805
  14:41:13 STEP 2105 H=2105:00 +CPU= 22.383
  14:41:54 STEP 2106 H=2106:00 +CPU= 36.898
  14:42:21 STEP 2107 H=2107:00 +CPU= 23.143
  14:42:48 STEP 2108 H=2108:00 +CPU= 24.223
  14:43:29 STEP 2109 H=2109:00 +CPU= 36.074
  14:43:57 STEP 2110 H=2110:00 +CPU= 24.512
  14:44:24 STEP 2111 H=2111:00 +CPU= 23.938
  14:45:11 STEP 2112 H=2112:00 +CPU= 41.826
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002112
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002112
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002112
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002088
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002088
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002088
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002100
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002100
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002100
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_87.zip
Uploading the intermediate file: upload_file_87.zip
Uploading trickle at timestep: 7603200
  14:45:37 STEP 2113 H=2113:00 +CPU= 23.140
  14:46:04 STEP 2114 H=2114:00 +CPU= 24.136
  14:46:43 STEP 2115 H=2115:00 +CPU= 35.396
  14:47:11 STEP 2116 H=2116:00 +CPU= 24.352
  14:47:36 STEP 2117 H=2117:00 +CPU= 22.879
  14:48:16 STEP 2118 H=2118:00 +CPU= 36.793
  14:48:42 STEP 2119 H=2119:00 +CPU= 22.549
  14:49:09 STEP 2120 H=2120:00 +CPU= 23.659
  14:49:50 STEP 2121 H=2121:00 +CPU= 36.760
  14:50:17 STEP 2122 H=2122:00 +CPU= 23.341
  14:50:42 STEP 2123 H=2123:00 +CPU= 22.901
  14:51:26 STEP 2124 H=2124:00 +CPU= 40.169
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002124
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002124
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002124
  14:51:52 STEP 2125 H=2125:00 +CPU= 23.573
  14:52:19 STEP 2126 H=2126:00 +CPU= 23.803
  14:52:58 STEP 2127 H=2127:00 +CPU= 35.759
  14:53:25 STEP 2128 H=2128:00 +CPU= 24.002
  14:53:51 STEP 2129 H=2129:00 +CPU= 22.971
  14:54:31 STEP 2130 H=2130:00 +CPU= 36.480
  14:54:58 STEP 2131 H=2131:00 +CPU= 23.319
  14:55:25 STEP 2132 H=2132:00 +CPU= 23.543
  14:56:05 STEP 2133 H=2133:00 +CPU= 36.235
  14:56:32 STEP 2134 H=2134:00 +CPU= 23.319
  14:56:59 STEP 2135 H=2135:00 +CPU= 24.207
  14:57:44 STEP 2136 H=2136:00 +CPU= 41.267
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002136
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002136
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002136
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002112
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002112
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002112
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002124
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002124
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002124
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_88.zip
Uploading the intermediate file: upload_file_88.zip
  14:58:10 STEP 2137 H=2137:00 +CPU= 22.928
Uploading trickle at timestep: 7689600
  14:58:37 STEP 2138 H=2138:00 +CPU= 24.335
  14:59:16 STEP 2139 H=2139:00 +CPU= 36.009
  14:59:42 STEP 2140 H=2140:00 +CPU= 23.998
  15:00:07 STEP 2141 H=2141:00 +CPU= 22.333
  15:00:48 STEP 2142 H=2142:00 +CPU= 36.881
  15:01:15 STEP 2143 H=2143:00 +CPU= 23.027
  15:01:42 STEP 2144 H=2144:00 +CPU= 24.035
  15:02:21 STEP 2145 H=2145:00 +CPU= 35.525
  15:02:48 STEP 2146 H=2146:00 +CPU= 24.101
  15:03:14 STEP 2147 H=2147:00 +CPU= 23.637
  15:03:58 STEP 2148 H=2148:00 +CPU= 40.040
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002148
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002148
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002148
  15:04:24 STEP 2149 H=2149:00 +CPU= 23.331
  15:04:52 STEP 2150 H=2150:00 +CPU= 24.251
  15:05:31 STEP 2151 H=2151:00 +CPU= 35.552
  15:05:58 STEP 2152 H=2152:00 +CPU= 24.431
  15:06:24 STEP 2153 H=2153:00 +CPU= 23.158
  15:07:05 STEP 2154 H=2154:00 +CPU= 36.644
  15:07:31 STEP 2155 H=2155:00 +CPU= 23.101
  15:07:58 STEP 2156 H=2156:00 +CPU= 23.910
  15:08:36 STEP 2157 H=2157:00 +CPU= 35.724
  15:09:02 STEP 2158 H=2158:00 +CPU= 23.711
  15:09:30 STEP 2159 H=2159:00 +CPU= 23.876
  15:10:17 STEP 2160 H=2160:00 +CPU= 42.609
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002160
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002160
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002160
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002136
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002136
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002136
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002148
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002148
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002148
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_89.zip
Uploading the intermediate file: upload_file_89.zip
Uploading trickle at timestep: 7776000
  15:10:44 STEP 2161 H=2161:00 +CPU= 23.312
  15:11:10 STEP 2162 H=2162:00 +CPU= 23.442
  15:11:51 STEP 2163 H=2163:00 +CPU= 36.157
  15:12:18 STEP 2164 H=2164:00 +CPU= 23.927
  15:12:43 STEP 2165 H=2165:00 +CPU= 22.628
  15:13:25 STEP 2166 H=2166:00 +CPU= 36.937
  15:13:51 STEP 2167 H=2167:00 +CPU= 23.299
  15:14:18 STEP 2168 H=2168:00 +CPU= 23.871
  15:14:57 STEP 2169 H=2169:00 +CPU= 35.620
  15:15:25 STEP 2170 H=2170:00 +CPU= 24.451
  15:15:52 STEP 2171 H=2171:00 +CPU= 23.310
  15:16:37 STEP 2172 H=2172:00 +CPU= 40.191
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002172
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002172
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002172
  15:17:04 STEP 2173 H=2173:00 +CPU= 23.041
  15:17:30 STEP 2174 H=2174:00 +CPU= 24.021
  15:18:11 STEP 2175 H=2175:00 +CPU= 35.634
  15:18:38 STEP 2176 H=2176:00 +CPU= 24.371
  15:19:04 STEP 2177 H=2177:00 +CPU= 22.965
  15:19:46 STEP 2178 H=2178:00 +CPU= 37.179
  15:20:12 STEP 2179 H=2179:00 +CPU= 23.500
  15:20:39 STEP 2180 H=2180:00 +CPU= 24.600
  15:21:20 STEP 2181 H=2181:00 +CPU= 36.305
  15:21:46 STEP 2182 H=2182:00 +CPU= 22.707
  15:22:15 STEP 2183 H=2183:00 +CPU= 24.263
  15:22:59 STEP 2184 H=2184:00 +CPU= 40.583
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002184
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002184
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002184
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002160
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002160
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002160
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002172
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002172
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002172
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_90.zip
Uploading the intermediate file: upload_file_90.zip
  15:23:26 STEP 2185 H=2185:00 +CPU= 23.156
Uploading trickle at timestep: 7862400
  15:23:53 STEP 2186 H=2186:00 +CPU= 23.326
  15:24:33 STEP 2187 H=2187:00 +CPU= 35.358
  15:25:00 STEP 2188 H=2188:00 +CPU= 23.769
  15:25:25 STEP 2189 H=2189:00 +CPU= 22.397
  15:26:07 STEP 2190 H=2190:00 +CPU= 36.229
  15:26:33 STEP 2191 H=2191:00 +CPU= 22.408
  15:27:01 STEP 2192 H=2192:00 +CPU= 23.772
  15:27:41 STEP 2193 H=2193:00 +CPU= 36.023
  15:28:08 STEP 2194 H=2194:00 +CPU= 24.043
  15:28:33 STEP 2195 H=2195:00 +CPU= 22.793
  15:29:18 STEP 2196 H=2196:00 +CPU= 40.496
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002196
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002196
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002196
  15:29:44 STEP 2197 H=2197:00 +CPU= 22.567
  15:30:11 STEP 2198 H=2198:00 +CPU= 23.954
  15:30:51 STEP 2199 H=2199:00 +CPU= 35.564
  15:31:17 STEP 2200 H=2200:00 +CPU= 23.580
  15:31:44 STEP 2201 H=2201:00 +CPU= 22.775
  15:32:26 STEP 2202 H=2202:00 +CPU= 37.174
  15:32:51 STEP 2203 H=2203:00 +CPU= 23.022
  15:33:17 STEP 2204 H=2204:00 +CPU= 23.802
  15:33:57 STEP 2205 H=2205:00 +CPU= 35.435
  15:34:23 STEP 2206 H=2206:00 +CPU= 23.622
  15:34:50 STEP 2207 H=2207:00 +CPU= 23.997
  15:35:38 STEP 2208 H=2208:00 +CPU= 41.528
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002208
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002208
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002208
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002184
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002184
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002184
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002196
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002196
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002196
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_91.zip
Uploading the intermediate file: upload_file_91.zip
  15:36:04 STEP 2209 H=2209:00 +CPU= 22.623
Uploading trickle at timestep: 7948800
  15:36:32 STEP 2210 H=2210:00 +CPU= 24.122
  15:37:13 STEP 2211 H=2211:00 +CPU= 36.051
  15:37:40 STEP 2212 H=2212:00 +CPU= 23.277
  15:38:04 STEP 2213 H=2213:00 +CPU= 23.003
  15:38:47 STEP 2214 H=2214:00 +CPU= 36.910
  15:39:14 STEP 2215 H=2215:00 +CPU= 22.930
  15:39:41 STEP 2216 H=2216:00 +CPU= 23.732
  15:40:23 STEP 2217 H=2217:00 +CPU= 36.481
  15:40:50 STEP 2218 H=2218:00 +CPU= 24.136
  15:41:16 STEP 2219 H=2219:00 +CPU= 22.703
  15:42:01 STEP 2220 H=2220:00 +CPU= 39.967
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002220
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002220
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002220
  15:42:27 STEP 2221 H=2221:00 +CPU= 22.640
  15:42:54 STEP 2222 H=2222:00 +CPU= 23.593
  15:43:33 STEP 2223 H=2223:00 +CPU= 35.161
  15:43:58 STEP 2224 H=2224:00 +CPU= 23.763
  15:44:24 STEP 2225 H=2225:00 +CPU= 23.371
  15:45:05 STEP 2226 H=2226:00 +CPU= 36.814
  15:45:33 STEP 2227 H=2227:00 +CPU= 23.136
  15:45:58 STEP 2228 H=2228:00 +CPU= 22.973
  15:46:39 STEP 2229 H=2229:00 +CPU= 35.746
  15:47:08 STEP 2230 H=2230:00 +CPU= 23.800
  15:47:34 STEP 2231 H=2231:00 +CPU= 23.272
  15:48:19 STEP 2232 H=2232:00 +CPU= 41.910
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002232
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002232
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002232
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002208
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002208
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002208
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002220
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002220
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002220
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_92.zip
Uploading the intermediate file: upload_file_92.zip
  15:48:44 STEP 2233 H=2233:00 +CPU= 23.353
Uploading trickle at timestep: 8035200
  15:49:12 STEP 2234 H=2234:00 +CPU= 24.311
  15:49:51 STEP 2235 H=2235:00 +CPU= 34.817
  15:50:16 STEP 2236 H=2236:00 +CPU= 23.441
  15:50:43 STEP 2237 H=2237:00 +CPU= 22.969
  15:51:23 STEP 2238 H=2238:00 +CPU= 36.744
  15:51:49 STEP 2239 H=2239:00 +CPU= 23.363
  15:52:15 STEP 2240 H=2240:00 +CPU= 23.472
  15:52:56 STEP 2241 H=2241:00 +CPU= 35.210
  15:53:22 STEP 2242 H=2242:00 +CPU= 23.810
  15:53:47 STEP 2243 H=2243:00 +CPU= 22.813
  15:54:31 STEP 2244 H=2244:00 +CPU= 39.520
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002244
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002244
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002244
  15:54:57 STEP 2245 H=2245:00 +CPU= 23.357
  15:55:21 STEP 2246 H=2246:00 +CPU= 23.561
  15:55:59 STEP 2247 H=2247:00 +CPU= 35.290
  15:56:26 STEP 2248 H=2248:00 +CPU= 23.487
  15:56:52 STEP 2249 H=2249:00 +CPU= 23.503
  15:57:35 STEP 2250 H=2250:00 +CPU= 36.625
  15:58:01 STEP 2251 H=2251:00 +CPU= 23.102
  15:58:27 STEP 2252 H=2252:00 +CPU= 23.469
  15:59:09 STEP 2253 H=2253:00 +CPU= 35.435
  15:59:35 STEP 2254 H=2254:00 +CPU= 23.616
  16:00:01 STEP 2255 H=2255:00 +CPU= 23.589
  16:00:47 STEP 2256 H=2256:00 +CPU= 42.676
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002256
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002256
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002256
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002232
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002232
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002232
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002244
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002244
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002244
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_93.zip
Uploading the intermediate file: upload_file_93.zip
Uploading trickle at timestep: 8121600
  16:01:15 STEP 2257 H=2257:00 +CPU= 23.479
  16:01:42 STEP 2258 H=2258:00 +CPU= 24.008
  16:02:23 STEP 2259 H=2259:00 +CPU= 35.679
  16:02:49 STEP 2260 H=2260:00 +CPU= 23.765
  16:03:15 STEP 2261 H=2261:00 +CPU= 22.842
  16:03:56 STEP 2262 H=2262:00 +CPU= 36.851
  16:04:23 STEP 2263 H=2263:00 +CPU= 23.402
  16:04:50 STEP 2264 H=2264:00 +CPU= 23.695
  16:05:31 STEP 2265 H=2265:00 +CPU= 35.631
  16:05:58 STEP 2266 H=2266:00 +CPU= 23.614
  16:06:23 STEP 2267 H=2267:00 +CPU= 22.547
  16:07:08 STEP 2268 H=2268:00 +CPU= 39.860
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002268
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002268
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002268
  16:07:33 STEP 2269 H=2269:00 +CPU= 22.715
  16:08:00 STEP 2270 H=2270:00 +CPU= 23.910
  16:08:39 STEP 2271 H=2271:00 +CPU= 35.577
  16:09:05 STEP 2272 H=2272:00 +CPU= 23.601
  16:09:32 STEP 2273 H=2273:00 +CPU= 22.766
  16:10:13 STEP 2274 H=2274:00 +CPU= 36.552
  16:10:41 STEP 2275 H=2275:00 +CPU= 23.203
  16:11:10 STEP 2276 H=2276:00 +CPU= 23.708
  16:11:50 STEP 2277 H=2277:00 +CPU= 35.796
  16:12:17 STEP 2278 H=2278:00 +CPU= 23.374
  16:12:44 STEP 2279 H=2279:00 +CPU= 24.766
  16:13:36 STEP 2280 H=2280:00 +CPU= 41.729
Moving to projects directory: /misc/boinc/slots/16/ICMGGhpi1+002280
Moving to projects directory: /misc/boinc/slots/16/ICMSHhpi1+002280
Moving to projects directory: /misc/boinc/slots/16/ICMUAhpi1+002280
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002256
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002256
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002256
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMGGhpi1+002268
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMSHhpi1+002268
Adding to the zip: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_12163713/ICMUAhpi1+002268
Zipping up the intermediate file: /misc/boinc/projects/climateprediction.net/oifs_43r3_ps_0624_2021050100_123_945_12163713_0_r1663596187_94.zip
Uploading the intermediate file: upload_file_94.zip
  16:14:02 STEP 2281 H=2281:00 +CPU= 22.583
Uploading trickle at timestep: 8208000
  16:14:28 STEP 2282 H=2282:00 +CPU= 24.050
  16:15:09 STEP 2283 H=2283:00 +CPU= 36.043
  16:15:35 STEP 2284 H=2284:00 +CPU= 23.624
  16:16:01 STEP 2285 H=2285:00 +CPU= 22.569
  16:16:41 STEP 2286 H=2286:00 +CPU= 36.953
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.429] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 3539MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.480] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.480] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.480] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc0c02], nsigs = 1
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.480] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.480] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3539 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.480] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.515] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.523] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.523] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CPG_DRV 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-CPG                  (1025) 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              CPG 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CPG_DIA 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161642:1669760202:77026.531] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                GPRH 
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/misc/boinc/slots/16/./master.exe' : sigcontextptr=0x7ffe98e4e910
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 21) with addr2line-cmd
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/misc/boinc/slots/16/./master.exe' 0x839273 0x839879 0x1dc9090 0x1cc0c02 0xcd7af6 0xc89cec 0xc2f59f 0xc2c638 0xbf17c8 0x12e9482 0xbbc1ce 0x57a664 0x55deec 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dc9090] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc0c02] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xcd7af6] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xc89cec] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xc2f59f] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xc2c638] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xbf17c8] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x12e9482] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbbc1ce] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x57a664] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x55deec] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x5471c9] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x546412] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x545e28] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x40708d] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x40220f] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x4021a2] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x1dca390] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x40206e] : ??() at ??:0
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.532] [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/misc/boinc/slots/16/./master.exe' : sigcontextptr=0x7ffe98e4e820
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 22) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/misc/boinc/slots/16/./master.exe' 0x839273 0x839879 0x821070 0x1dc9090 0x1cc0c02 0xcd7af6 0xc89cec 0xc2f59f 0xc2c638 0xbf17c8 0x12e9482 0xbbc1ce 0x57a664 0x55deec 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x821070] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dc9090] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc0c02] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0xcd7af6] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0xc89cec] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xc2f59f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xc2c638] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xbf17c8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x12e9482] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbbc1ce] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x57a664] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x55deec] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.900] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:minstrel:1:1:1719:1719] [20221129:161643:1669760203:77027.900] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
master.exe         0000000001CE916B  Unknown               Unknown  Unknown
master.exe         000000000081FFF1  Unknown               Unknown  Unknown
master.exe         0000000001DC9090  Unknown               Unknown  Unknown
master.exe         0000000001CC0C02  Unknown               Unknown  Unknown
master.exe         0000000000CD7AF6  gprh_                     128  gprh.F90
master.exe         0000000000C89CEC  cpg_dia_                  600  cpg_dia.F90
master.exe         0000000000C2F59F  cpg_                      718  cpg.F90
master.exe         0000000000C2C638  cpg_drv_                  351  cpg_drv.F90
master.exe         0000000000BF17C8  gp_model_                 419  gp_model.F90
master.exe         00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
master.exe         0000000000BBC1CE  scan2m_                   535  scan2m.F90
master.exe         000000000057A664  stepo_                    327  stepo.F90
master.exe         000000000055DEEC  cnt4_                    1133  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:05:57 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 8,208,000 59,516 0.0073
09 May 2023 15:05:57 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 8,121,600 58,837 0.0072
09 May 2023 15:05:57 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 8,035,200 58,175 0.0072
09 May 2023 15:05:57 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,948,800 57,499 0.0072
09 May 2023 15:05:25 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,862,400 56,827 0.0072
09 May 2023 15:05:25 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,776,000 56,146 0.0072
09 May 2023 15:05:25 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,689,600 55,474 0.0072
09 May 2023 15:05:25 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,603,200 54,795 0.0072
09 May 2023 15:04:46 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,516,800 54,118 0.0072
09 May 2023 15:04:46 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,430,400 53,443 0.0072
09 May 2023 15:04:46 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,344,000 52,751 0.0072
09 May 2023 15:04:46 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,257,600 52,062 0.0072
09 May 2023 15:04:46 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,171,200 51,384 0.0072
09 May 2023 15:04:10 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 7,084,800 50,733 0.0072
09 May 2023 15:04:10 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 6,998,400 50,064 0.0072
09 May 2023 15:04:10 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 6,912,000 49,401 0.0071
09 May 2023 15:04:10 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 6,825,600 48,747 0.0071
09 May 2023 15:04:10 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 6,739,200 48,091 0.0071
09 May 2023 15:04:10 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 6,652,800 47,405 0.0071
09 May 2023 15:03:34 1378058 22245657 oifs_43r3_ps_0624_2021050100_123_945_12163713_0 6,566,400 46,830 0.0071


©2024 cpdn.org