climateprediction.net home page
Task 22302317

Task 22302317

Name oifs_43r3_ps_0610_2014050100_123_983_12200254_1
Workunit 12200254
Created 23 Jan 2023, 10:49:17 UTC
Sent 23 Jan 2023, 11:28:49 UTC
Report deadline 24 Mar 2023, 11:28:49 UTC
Received 25 Jan 2023, 6:14:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1499708
Run time 12 hours 49 min 34 sec
CPU time 12 hours 17 min 46 sec
Validate state Invalid
Credit 8,225.28
Device peak FLOPS 7.15 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,508.27 MB
Peak swap size 4,978.11 MB
Peak disk usage 7,409.20 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:11822:11822] [20230125:033028:1674613828:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:31:11 STEP 1872 H=1872:00 +CPU= 31.115
  03:31:30 STEP 1873 H=1873:00 +CPU= 16.912
  03:31:53 STEP 1874 H=1874:00 +CPU= 17.920
  03:32:18 STEP 1875 H=1875:00 +CPU= 23.580
  03:32:32 STEP 1876 H=1876:00 +CPU= 14.046
  03:32:46 STEP 1877 H=1877:00 +CPU= 13.690
  03:33:08 STEP 1878 H=1878:00 +CPU= 21.820
  03:33:22 STEP 1879 H=1879:00 +CPU= 13.557
  03:33:36 STEP 1880 H=1880:00 +CPU= 13.820
  03:33:58 STEP 1881 H=1881:00 +CPU= 21.313
  03:34:11 STEP 1882 H=1882:00 +CPU= 12.502
  03:34:22 STEP 1883 H=1883:00 +CPU= 11.617
  03:34:44 STEP 1884 H=1884:00 +CPU= 20.804
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001884
  03:34:56 STEP 1885 H=1885:00 +CPU= 11.727
  03:35:10 STEP 1886 H=1886:00 +CPU= 14.417
  03:35:36 STEP 1887 H=1887:00 +CPU= 25.063
  03:35:53 STEP 1888 H=1888:00 +CPU= 16.460
  03:36:09 STEP 1889 H=1889:00 +CPU= 15.570
  03:36:35 STEP 1890 H=1890:00 +CPU= 25.001
  03:36:48 STEP 1891 H=1891:00 +CPU= 12.756
  03:37:02 STEP 1892 H=1892:00 +CPU= 13.280
  03:37:23 STEP 1893 H=1893:00 +CPU= 20.762
  03:37:37 STEP 1894 H=1894:00 +CPU= 13.307
  03:37:55 STEP 1895 H=1895:00 +CPU= 16.541
  03:38:41 STEP 1896 H=1896:00 +CPU= 30.156
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_78.zip
Uploading the intermediate file: upload_file_78.zip
  03:38:58 STEP 1897 H=1897:00 +CPU= 16.307
  03:39:17 STEP 1898 H=1898:00 +CPU= 17.995
  03:39:43 STEP 1899 H=1899:00 +CPU= 25.308
  03:40:00 STEP 1900 H=1900:00 +CPU= 16.088
  03:40:15 STEP 1901 H=1901:00 +CPU= 14.902
  03:40:40 STEP 1902 H=1902:00 +CPU= 23.694
  03:40:54 STEP 1903 H=1903:00 +CPU= 14.410
  03:41:09 STEP 1904 H=1904:00 +CPU= 14.677
  03:41:34 STEP 1905 H=1905:00 +CPU= 23.940
  03:41:50 STEP 1906 H=1906:00 +CPU= 15.240
  03:42:05 STEP 1907 H=1907:00 +CPU= 14.495
  03:42:31 STEP 1908 H=1908:00 +CPU= 25.223
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001908
  03:42:45 STEP 1909 H=1909:00 +CPU= 14.372
  03:43:00 STEP 1910 H=1910:00 +CPU= 14.564
  03:43:23 STEP 1911 H=1911:00 +CPU= 21.908
  03:43:36 STEP 1912 H=1912:00 +CPU= 13.203
  03:43:49 STEP 1913 H=1913:00 +CPU= 12.640
  03:44:10 STEP 1914 H=1914:00 +CPU= 21.077
  03:44:25 STEP 1915 H=1915:00 +CPU= 14.383
  03:44:41 STEP 1916 H=1916:00 +CPU= 14.905
  03:45:02 STEP 1917 H=1917:00 +CPU= 21.319
  03:45:15 STEP 1918 H=1918:00 +CPU= 12.877
  03:45:29 STEP 1919 H=1919:00 +CPU= 12.971
  03:45:57 STEP 1920 H=1920:00 +CPU= 24.241
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_79.zip
Uploading the intermediate file: upload_file_79.zip
  03:46:13 STEP 1921 H=1921:00 +CPU= 15.982
  03:46:29 STEP 1922 H=1922:00 +CPU= 15.166
  03:46:53 STEP 1923 H=1923:00 +CPU= 23.867
  03:47:10 STEP 1924 H=1924:00 +CPU= 16.318
  03:47:27 STEP 1925 H=1925:00 +CPU= 16.258
  03:47:50 STEP 1926 H=1926:00 +CPU= 22.232
  03:48:03 STEP 1927 H=1927:00 +CPU= 12.763
  03:48:16 STEP 1928 H=1928:00 +CPU= 13.257
  03:48:38 STEP 1929 H=1929:00 +CPU= 20.943
  03:48:55 STEP 1930 H=1930:00 +CPU= 16.879
  03:49:11 STEP 1931 H=1931:00 +CPU= 15.485
  03:49:39 STEP 1932 H=1932:00 +CPU= 27.250
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001932
  03:49:55 STEP 1933 H=1933:00 +CPU= 15.396
  03:50:12 STEP 1934 H=1934:00 +CPU= 15.839
  03:50:37 STEP 1935 H=1935:00 +CPU= 24.608
  03:50:54 STEP 1936 H=1936:00 +CPU= 16.274
  03:51:11 STEP 1937 H=1937:00 +CPU= 16.773
  03:51:36 STEP 1938 H=1938:00 +CPU= 24.294
  03:51:50 STEP 1939 H=1939:00 +CPU= 13.058
  03:52:03 STEP 1940 H=1940:00 +CPU= 13.252
  03:52:25 STEP 1941 H=1941:00 +CPU= 21.095
  03:52:42 STEP 1942 H=1942:00 +CPU= 16.002
  03:52:59 STEP 1943 H=1943:00 +CPU= 16.837
  03:53:31 STEP 1944 H=1944:00 +CPU= 30.099
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_80.zip
Uploading the intermediate file: upload_file_80.zip
  03:53:45 STEP 1945 H=1945:00 +CPU= 13.359
  03:53:59 STEP 1946 H=1946:00 +CPU= 13.888
  03:54:21 STEP 1947 H=1947:00 +CPU= 21.855
  03:54:37 STEP 1948 H=1948:00 +CPU= 14.939
  03:54:55 STEP 1949 H=1949:00 +CPU= 16.945
  03:55:19 STEP 1950 H=1950:00 +CPU= 23.563
  03:55:32 STEP 1951 H=1951:00 +CPU= 13.130
  03:55:47 STEP 1952 H=1952:00 +CPU= 13.890
  03:56:08 STEP 1953 H=1953:00 +CPU= 20.677
  03:56:24 STEP 1954 H=1954:00 +CPU= 15.838
  03:56:39 STEP 1955 H=1955:00 +CPU= 14.153
  03:57:03 STEP 1956 H=1956:00 +CPU= 23.476
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001956
  03:57:16 STEP 1957 H=1957:00 +CPU= 12.824
  03:57:30 STEP 1958 H=1958:00 +CPU= 13.229
  03:57:51 STEP 1959 H=1959:00 +CPU= 20.458
  03:58:06 STEP 1960 H=1960:00 +CPU= 14.512
  03:58:22 STEP 1961 H=1961:00 +CPU= 16.139
  03:58:46 STEP 1962 H=1962:00 +CPU= 23.423
  03:59:00 STEP 1963 H=1963:00 +CPU= 13.597
  03:59:14 STEP 1964 H=1964:00 +CPU= 13.570
  03:59:36 STEP 1965 H=1965:00 +CPU= 21.420
  03:59:50 STEP 1966 H=1966:00 +CPU= 13.813
  04:00:07 STEP 1967 H=1967:00 +CPU= 15.946
  04:00:39 STEP 1968 H=1968:00 +CPU= 30.564
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_81.zip
Uploading the intermediate file: upload_file_81.zip
  04:00:56 STEP 1969 H=1969:00 +CPU= 15.816
  04:01:13 STEP 1970 H=1970:00 +CPU= 16.421
  04:01:41 STEP 1971 H=1971:00 +CPU= 26.604
  04:01:59 STEP 1972 H=1972:00 +CPU= 17.516
  04:02:17 STEP 1973 H=1973:00 +CPU= 17.113
  04:02:45 STEP 1974 H=1974:00 +CPU= 27.219
  04:03:03 STEP 1975 H=1975:00 +CPU= 16.598
  04:03:20 STEP 1976 H=1976:00 +CPU= 16.886
  04:03:47 STEP 1977 H=1977:00 +CPU= 25.812
  04:04:04 STEP 1978 H=1978:00 +CPU= 16.063
  04:04:20 STEP 1979 H=1979:00 +CPU= 15.879
  04:04:51 STEP 1980 H=1980:00 +CPU= 29.524
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001980
  04:05:09 STEP 1981 H=1981:00 +CPU= 16.970
  04:05:26 STEP 1982 H=1982:00 +CPU= 16.948
  04:05:53 STEP 1983 H=1983:00 +CPU= 25.293
  04:06:10 STEP 1984 H=1984:00 +CPU= 16.559
  04:06:27 STEP 1985 H=1985:00 +CPU= 16.526
  04:06:54 STEP 1986 H=1986:00 +CPU= 26.060
  04:07:11 STEP 1987 H=1987:00 +CPU= 15.977
  04:07:27 STEP 1988 H=1988:00 +CPU= 16.152
  04:07:54 STEP 1989 H=1989:00 +CPU= 25.398
  04:08:11 STEP 1990 H=1990:00 +CPU= 16.559
  04:08:28 STEP 1991 H=1991:00 +CPU= 16.068
  04:08:58 STEP 1992 H=1992:00 +CPU= 29.049
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_82.zip
Uploading the intermediate file: upload_file_82.zip
  04:09:14 STEP 1993 H=1993:00 +CPU= 15.116
  04:09:30 STEP 1994 H=1994:00 +CPU= 15.702
  04:09:56 STEP 1995 H=1995:00 +CPU= 24.885
  04:10:13 STEP 1996 H=1996:00 +CPU= 17.223
  04:10:31 STEP 1997 H=1997:00 +CPU= 16.906
  04:10:57 STEP 1998 H=1998:00 +CPU= 25.176
  04:11:12 STEP 1999 H=1999:00 +CPU= 15.090
  04:11:28 STEP 2000 H=2000:00 +CPU= 15.452
  04:11:55 STEP 2001 H=2001:00 +CPU= 26.055
  04:12:13 STEP 2002 H=2002:00 +CPU= 17.323
  04:12:29 STEP 2003 H=2003:00 +CPU= 15.287
  04:12:56 STEP 2004 H=2004:00 +CPU= 26.588
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002004
  04:13:11 STEP 2005 H=2005:00 +CPU= 14.737
  04:13:28 STEP 2006 H=2006:00 +CPU= 16.201
  04:13:54 STEP 2007 H=2007:00 +CPU= 25.342
  04:14:11 STEP 2008 H=2008:00 +CPU= 15.947
  04:14:27 STEP 2009 H=2009:00 +CPU= 15.470
  04:14:53 STEP 2010 H=2010:00 +CPU= 25.112
  04:15:09 STEP 2011 H=2011:00 +CPU= 15.563
  04:15:27 STEP 2012 H=2012:00 +CPU= 17.614
  04:15:55 STEP 2013 H=2013:00 +CPU= 26.849
  04:16:11 STEP 2014 H=2014:00 +CPU= 15.629
  04:16:27 STEP 2015 H=2015:00 +CPU= 15.230
  04:16:57 STEP 2016 H=2016:00 +CPU= 29.242
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_83.zip
Uploading the intermediate file: upload_file_83.zip
  04:17:13 STEP 2017 H=2017:00 +CPU= 15.819
  04:17:30 STEP 2018 H=2018:00 +CPU= 16.012
  04:17:56 STEP 2019 H=2019:00 +CPU= 24.904
  04:18:12 STEP 2020 H=2020:00 +CPU= 15.525
  04:18:28 STEP 2021 H=2021:00 +CPU= 15.212
  04:18:52 STEP 2022 H=2022:00 +CPU= 24.230
  04:19:07 STEP 2023 H=2023:00 +CPU= 13.889
  04:19:21 STEP 2024 H=2024:00 +CPU= 13.669
  04:19:42 STEP 2025 H=2025:00 +CPU= 20.735
  04:19:55 STEP 2026 H=2026:00 +CPU= 13.259
  04:20:09 STEP 2027 H=2027:00 +CPU= 12.704
  04:20:36 STEP 2028 H=2028:00 +CPU= 26.795
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002028
  04:20:52 STEP 2029 H=2029:00 +CPU= 15.195
  04:21:09 STEP 2030 H=2030:00 +CPU= 16.118
  04:21:29 STEP 2031 H=2031:00 +CPU= 19.829
  04:21:41 STEP 2032 H=2032:00 +CPU= 11.933
  04:21:53 STEP 2033 H=2033:00 +CPU= 11.972
  04:22:13 STEP 2034 H=2034:00 +CPU= 19.395
  04:22:29 STEP 2035 H=2035:00 +CPU= 15.531
  04:22:43 STEP 2036 H=2036:00 +CPU= 13.284
  04:23:03 STEP 2037 H=2037:00 +CPU= 19.414
  04:23:20 STEP 2038 H=2038:00 +CPU= 12.249
  04:23:32 STEP 2039 H=2039:00 +CPU= 12.345
  04:24:02 STEP 2040 H=2040:00 +CPU= 26.808
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_84.zip
Uploading the intermediate file: upload_file_84.zip
  04:24:16 STEP 2041 H=2041:00 +CPU= 13.907
  04:24:30 STEP 2042 H=2042:00 +CPU= 14.015
  04:24:53 STEP 2043 H=2043:00 +CPU= 22.051
  04:25:08 STEP 2044 H=2044:00 +CPU= 14.240
  04:25:23 STEP 2045 H=2045:00 +CPU= 14.525
  04:25:45 STEP 2046 H=2046:00 +CPU= 22.116
  04:25:57 STEP 2047 H=2047:00 +CPU= 11.921
  04:26:10 STEP 2048 H=2048:00 +CPU= 12.278
  04:26:29 STEP 2049 H=2049:00 +CPU= 19.099
  04:26:42 STEP 2050 H=2050:00 +CPU= 12.483
  04:26:55 STEP 2051 H=2051:00 +CPU= 12.960
  04:27:17 STEP 2052 H=2052:00 +CPU= 21.651
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002052
  04:27:29 STEP 2053 H=2053:00 +CPU= 11.832
  04:27:42 STEP 2054 H=2054:00 +CPU= 12.606
  04:28:04 STEP 2055 H=2055:00 +CPU= 21.610
  04:28:19 STEP 2056 H=2056:00 +CPU= 14.471
  04:28:33 STEP 2057 H=2057:00 +CPU= 14.016
  04:28:57 STEP 2058 H=2058:00 +CPU= 22.605
  04:29:10 STEP 2059 H=2059:00 +CPU= 13.383
  04:29:25 STEP 2060 H=2060:00 +CPU= 14.202
  04:29:46 STEP 2061 H=2061:00 +CPU= 20.940
  04:29:59 STEP 2062 H=2062:00 +CPU= 12.276
  04:30:11 STEP 2063 H=2063:00 +CPU= 12.230
  04:30:35 STEP 2064 H=2064:00 +CPU= 22.395
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_85.zip
Uploading the intermediate file: upload_file_85.zip
  04:30:48 STEP 2065 H=2065:00 +CPU= 12.371
  04:31:03 STEP 2066 H=2066:00 +CPU= 14.198
  04:31:27 STEP 2067 H=2067:00 +CPU= 23.305
  04:31:42 STEP 2068 H=2068:00 +CPU= 14.649
  04:31:57 STEP 2069 H=2069:00 +CPU= 14.480
  04:32:21 STEP 2070 H=2070:00 +CPU= 23.591
  04:32:36 STEP 2071 H=2071:00 +CPU= 14.426
  04:32:52 STEP 2072 H=2072:00 +CPU= 14.981
  04:33:17 STEP 2073 H=2073:00 +CPU= 24.091
  04:33:33 STEP 2074 H=2074:00 +CPU= 15.450
  04:33:49 STEP 2075 H=2075:00 +CPU= 15.348
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0610
(argv4) batchid: 983
(argv5) wuid: 12200254
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/22
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/22/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/22/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ca16c99ed4fc05b622ad3edd54bfd939 to: /var/lib/boinc/slots/22/oifs_43r3_ps_0610_2014050100_123_983_12200254.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/22/oifs_43r3_ps_0610_2014050100_123_983_12200254.zip
ic_ancil_file: ic_ancil_12200254
ifsdata_file: ifsdata_12200254
climate_data_file: clim_data_12200254
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4fa1867a339f43ddea47b8a0939bcf72 to: /var/lib/boinc/slots/22/ic_ancil_12200254.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/22/ic_ancil_12200254.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_19bf4f0ca2f9cb95c91631f4d49697c7 to: /var/lib/boinc/slots/22/ifsdata/ifsdata_12200254.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/22/ifsdata/ifsdata_12200254.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/22/159l_2/clim_data_12200254.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/22/159l_2/clim_data_12200254.zip
Checking for progress XML file: /var/lib/boinc/slots/22/progress_file_12200254.xml
Opened progress file ok : /var/lib/boinc/slots/22/progress_file_12200254.xml
-- Model is restarting --
Adjusting last_iter, 2076, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/22/progress_file_12200254.xml
last_cpu_time: 40372
upload_file_number: 86
last_iter: 2063
last_upload: 7430400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296
The child process has been launched with process id: 13430
Executing the command: /var/lib/boinc/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:13430:13430] [20230125:051633:1674620193:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:17:07 STEP 2064 H=2064:00 +CPU= 28.306
  05:17:23 STEP 2065 H=2065:00 +CPU= 14.570
  05:17:37 STEP 2066 H=2066:00 +CPU= 13.804
  05:17:59 STEP 2067 H=2067:00 +CPU= 21.662
  05:18:12 STEP 2068 H=2068:00 +CPU= 12.089
  05:18:23 STEP 2069 H=2069:00 +CPU= 11.249
  05:18:42 STEP 2070 H=2070:00 +CPU= 18.980
  05:18:55 STEP 2071 H=2071:00 +CPU= 12.242
  05:19:10 STEP 2072 H=2072:00 +CPU= 14.792
  05:19:32 STEP 2073 H=2073:00 +CPU= 21.245
  05:19:45 STEP 2074 H=2074:00 +CPU= 12.900
  05:19:57 STEP 2075 H=2075:00 +CPU= 11.951
  05:20:20 STEP 2076 H=2076:00 +CPU= 21.952
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002076
  05:20:32 STEP 2077 H=2077:00 +CPU= 12.261
  05:20:45 STEP 2078 H=2078:00 +CPU= 12.258
  05:21:04 STEP 2079 H=2079:00 +CPU= 19.183
  05:21:17 STEP 2080 H=2080:00 +CPU= 12.057
  05:21:29 STEP 2081 H=2081:00 +CPU= 11.678
  05:21:50 STEP 2082 H=2082:00 +CPU= 21.116
  05:22:04 STEP 2083 H=2083:00 +CPU= 13.463
  05:22:18 STEP 2084 H=2084:00 +CPU= 13.744
  05:22:41 STEP 2085 H=2085:00 +CPU= 22.045
  05:22:56 STEP 2086 H=2086:00 +CPU= 14.856
  05:23:11 STEP 2087 H=2087:00 +CPU= 14.427
  05:23:38 STEP 2088 H=2088:00 +CPU= 26.340
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_86.zip
Uploading the intermediate file: upload_file_86.zip
  05:23:54 STEP 2089 H=2089:00 +CPU= 14.768
  05:24:09 STEP 2090 H=2090:00 +CPU= 14.837
  05:24:29 STEP 2091 H=2091:00 +CPU= 19.979
  05:24:41 STEP 2092 H=2092:00 +CPU= 11.798
  05:24:53 STEP 2093 H=2093:00 +CPU= 11.445
  05:25:13 STEP 2094 H=2094:00 +CPU= 19.219
  05:25:28 STEP 2095 H=2095:00 +CPU= 13.626
  05:25:44 STEP 2096 H=2096:00 +CPU= 15.741
  05:26:07 STEP 2097 H=2097:00 +CPU= 22.736
  05:26:22 STEP 2098 H=2098:00 +CPU= 14.366
  05:26:36 STEP 2099 H=2099:00 +CPU= 13.704
  05:27:02 STEP 2100 H=2100:00 +CPU= 24.987
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002100
  05:27:17 STEP 2101 H=2101:00 +CPU= 14.809
  05:27:33 STEP 2102 H=2102:00 +CPU= 15.429
  05:27:58 STEP 2103 H=2103:00 +CPU= 24.205
  05:28:14 STEP 2104 H=2104:00 +CPU= 15.701
  05:28:28 STEP 2105 H=2105:00 +CPU= 13.706
  05:28:51 STEP 2106 H=2106:00 +CPU= 22.056
  05:29:05 STEP 2107 H=2107:00 +CPU= 13.384
  05:29:18 STEP 2108 H=2108:00 +CPU= 12.646
  05:29:37 STEP 2109 H=2109:00 +CPU= 18.792
  05:29:49 STEP 2110 H=2110:00 +CPU= 12.002
  05:30:01 STEP 2111 H=2111:00 +CPU= 11.760
  05:30:26 STEP 2112 H=2112:00 +CPU= 24.705
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_87.zip
Uploading the intermediate file: upload_file_87.zip
  05:30:41 STEP 2113 H=2113:00 +CPU= 13.737
  05:30:54 STEP 2114 H=2114:00 +CPU= 13.510
  05:31:17 STEP 2115 H=2115:00 +CPU= 21.557
  05:31:29 STEP 2116 H=2116:00 +CPU= 12.643
  05:31:41 STEP 2117 H=2117:00 +CPU= 11.680
  05:32:01 STEP 2118 H=2118:00 +CPU= 19.351
  05:32:13 STEP 2119 H=2119:00 +CPU= 11.681
  05:32:27 STEP 2120 H=2120:00 +CPU= 13.523
  05:32:49 STEP 2121 H=2121:00 +CPU= 21.995
  05:33:04 STEP 2122 H=2122:00 +CPU= 13.921
  05:33:18 STEP 2123 H=2123:00 +CPU= 13.561
  05:33:43 STEP 2124 H=2124:00 +CPU= 25.159
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002124
  05:33:58 STEP 2125 H=2125:00 +CPU= 13.932
  05:34:13 STEP 2126 H=2126:00 +CPU= 14.528
  05:34:36 STEP 2127 H=2127:00 +CPU= 22.745
  05:34:51 STEP 2128 H=2128:00 +CPU= 14.639
  05:35:06 STEP 2129 H=2129:00 +CPU= 14.239
  05:35:31 STEP 2130 H=2130:00 +CPU= 23.600
  05:35:46 STEP 2131 H=2131:00 +CPU= 14.451
  05:36:01 STEP 2132 H=2132:00 +CPU= 14.828
  05:36:25 STEP 2133 H=2133:00 +CPU= 22.649
  05:36:40 STEP 2134 H=2134:00 +CPU= 15.010
  05:36:56 STEP 2135 H=2135:00 +CPU= 14.871
  05:37:24 STEP 2136 H=2136:00 +CPU= 26.931
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_88.zip
Uploading the intermediate file: upload_file_88.zip
  05:37:39 STEP 2137 H=2137:00 +CPU= 14.467
  05:37:54 STEP 2138 H=2138:00 +CPU= 14.973
  05:38:18 STEP 2139 H=2139:00 +CPU= 23.223
  05:38:34 STEP 2140 H=2140:00 +CPU= 15.104
  05:38:49 STEP 2141 H=2141:00 +CPU= 14.568
  05:39:14 STEP 2142 H=2142:00 +CPU= 23.740
  05:39:29 STEP 2143 H=2143:00 +CPU= 14.450
  05:39:45 STEP 2144 H=2144:00 +CPU= 14.752
  05:40:09 STEP 2145 H=2145:00 +CPU= 23.302
  05:40:25 STEP 2146 H=2146:00 +CPU= 15.077
  05:40:40 STEP 2147 H=2147:00 +CPU= 14.558
  05:41:07 STEP 2148 H=2148:00 +CPU= 26.116
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002148
  05:41:22 STEP 2149 H=2149:00 +CPU= 14.412
  05:41:37 STEP 2150 H=2150:00 +CPU= 14.855
  05:42:01 STEP 2151 H=2151:00 +CPU= 23.275
  05:42:17 STEP 2152 H=2152:00 +CPU= 14.954
  05:42:32 STEP 2153 H=2153:00 +CPU= 14.600
  05:42:57 STEP 2154 H=2154:00 +CPU= 23.784
  05:43:12 STEP 2155 H=2155:00 +CPU= 14.600
  05:43:27 STEP 2156 H=2156:00 +CPU= 14.810
  05:43:51 STEP 2157 H=2157:00 +CPU= 23.196
  05:44:06 STEP 2158 H=2158:00 +CPU= 15.003
  05:44:22 STEP 2159 H=2159:00 +CPU= 14.897
  05:44:51 STEP 2160 H=2160:00 +CPU= 27.540
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_89.zip
Uploading the intermediate file: upload_file_89.zip
  05:45:06 STEP 2161 H=2161:00 +CPU= 14.401
  05:45:22 STEP 2162 H=2162:00 +CPU= 15.065
  05:45:46 STEP 2163 H=2163:00 +CPU= 23.346
  05:46:01 STEP 2164 H=2164:00 +CPU= 14.871
  05:46:17 STEP 2165 H=2165:00 +CPU= 14.726
  05:46:41 STEP 2166 H=2166:00 +CPU= 23.884
  05:46:57 STEP 2167 H=2167:00 +CPU= 14.799
  05:47:13 STEP 2168 H=2168:00 +CPU= 15.000
  05:47:37 STEP 2169 H=2169:00 +CPU= 23.429
  05:47:53 STEP 2170 H=2170:00 +CPU= 15.391
  05:48:08 STEP 2171 H=2171:00 +CPU= 14.568
  05:48:36 STEP 2172 H=2172:00 +CPU= 26.354
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002172
  05:48:50 STEP 2173 H=2173:00 +CPU= 14.406
  05:49:06 STEP 2174 H=2174:00 +CPU= 15.006
  05:49:30 STEP 2175 H=2175:00 +CPU= 23.521
  05:49:46 STEP 2176 H=2176:00 +CPU= 15.245
  05:50:01 STEP 2177 H=2177:00 +CPU= 14.435
  05:50:26 STEP 2178 H=2178:00 +CPU= 24.096
  05:50:41 STEP 2179 H=2179:00 +CPU= 14.759
  05:50:57 STEP 2180 H=2180:00 +CPU= 14.931
  05:51:21 STEP 2181 H=2181:00 +CPU= 23.209
  05:51:36 STEP 2182 H=2182:00 +CPU= 15.035
  05:51:52 STEP 2183 H=2183:00 +CPU= 15.093
  05:52:20 STEP 2184 H=2184:00 +CPU= 27.318
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_90.zip
Uploading the intermediate file: upload_file_90.zip
  05:52:35 STEP 2185 H=2185:00 +CPU= 14.607
  05:52:51 STEP 2186 H=2186:00 +CPU= 14.966
  05:53:15 STEP 2187 H=2187:00 +CPU= 23.511
  05:53:31 STEP 2188 H=2188:00 +CPU= 15.017
  05:53:46 STEP 2189 H=2189:00 +CPU= 14.630
  05:54:11 STEP 2190 H=2190:00 +CPU= 24.146
  05:54:27 STEP 2191 H=2191:00 +CPU= 14.875
  05:54:42 STEP 2192 H=2192:00 +CPU= 15.119
  05:55:07 STEP 2193 H=2193:00 +CPU= 23.641
  05:55:23 STEP 2194 H=2194:00 +CPU= 15.027
  05:55:38 STEP 2195 H=2195:00 +CPU= 14.562
  05:56:06 STEP 2196 H=2196:00 +CPU= 26.485
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002196
  05:56:21 STEP 2197 H=2197:00 +CPU= 14.578
  05:56:37 STEP 2198 H=2198:00 +CPU= 15.267
  05:57:01 STEP 2199 H=2199:00 +CPU= 23.457
  05:57:17 STEP 2200 H=2200:00 +CPU= 15.153
  05:57:32 STEP 2201 H=2201:00 +CPU= 14.658
  05:57:57 STEP 2202 H=2202:00 +CPU= 23.943
  05:58:12 STEP 2203 H=2203:00 +CPU= 14.664
  05:58:28 STEP 2204 H=2204:00 +CPU= 15.068
  05:58:53 STEP 2205 H=2205:00 +CPU= 23.595
  05:59:08 STEP 2206 H=2206:00 +CPU= 15.116
  05:59:24 STEP 2207 H=2207:00 +CPU= 14.931
  05:59:53 STEP 2208 H=2208:00 +CPU= 27.787
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_91.zip
Uploading the intermediate file: upload_file_91.zip
  06:00:08 STEP 2209 H=2209:00 +CPU= 14.480
  06:00:23 STEP 2210 H=2210:00 +CPU= 15.129
  06:00:48 STEP 2211 H=2211:00 +CPU= 23.488
  06:01:03 STEP 2212 H=2212:00 +CPU= 15.067
  06:01:18 STEP 2213 H=2213:00 +CPU= 14.675
  06:01:45 STEP 2214 H=2214:00 +CPU= 25.408
  06:02:01 STEP 2215 H=2215:00 +CPU= 15.287
  06:02:17 STEP 2216 H=2216:00 +CPU= 16.144
  06:02:43 STEP 2217 H=2217:00 +CPU= 24.733
  06:02:58 STEP 2218 H=2218:00 +CPU= 15.270
  06:03:13 STEP 2219 H=2219:00 +CPU= 14.264
  06:03:37 STEP 2220 H=2220:00 +CPU= 23.038
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002220
  06:03:48 STEP 2221 H=2221:00 +CPU= 11.136
  06:04:00 STEP 2222 H=2222:00 +CPU= 11.357
  06:04:18 STEP 2223 H=2223:00 +CPU= 18.045
  06:04:30 STEP 2224 H=2224:00 +CPU= 11.488
  06:04:43 STEP 2225 H=2225:00 +CPU= 12.808
  06:05:07 STEP 2226 H=2226:00 +CPU= 23.965
  06:05:21 STEP 2227 H=2227:00 +CPU= 13.149
  06:05:34 STEP 2228 H=2228:00 +CPU= 12.886
  06:05:54 STEP 2229 H=2229:00 +CPU= 19.691
  06:06:06 STEP 2230 H=2230:00 +CPU= 11.828
  06:06:18 STEP 2231 H=2231:00 +CPU= 11.847
  06:06:40 STEP 2232 H=2232:00 +CPU= 21.717
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_92.zip
Uploading the intermediate file: upload_file_92.zip
  06:06:52 STEP 2233 H=2233:00 +CPU= 11.418
  06:07:07 STEP 2234 H=2234:00 +CPU= 14.522
  06:07:32 STEP 2235 H=2235:00 +CPU= 23.582
  06:07:47 STEP 2236 H=2236:00 +CPU= 14.622
  06:08:02 STEP 2237 H=2237:00 +CPU= 14.425
  06:08:27 STEP 2238 H=2238:00 +CPU= 24.310
  06:08:42 STEP 2239 H=2239:00 +CPU= 14.529
  06:08:57 STEP 2240 H=2240:00 +CPU= 14.674
  06:09:20 STEP 2241 H=2241:00 +CPU= 21.968
  06:09:32 STEP 2242 H=2242:00 +CPU= 12.282
  06:09:45 STEP 2243 H=2243:00 +CPU= 12.743
  06:10:05 STEP 2244 H=2244:00 +CPU= 20.027
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002244
  06:10:17 STEP 2245 H=2245:00 +CPU= 11.314
  06:10:29 STEP 2246 H=2246:00 +CPU= 11.505
  06:10:47 STEP 2247 H=2247:00 +CPU= 17.885
  06:11:01 STEP 2248 H=2248:00 +CPU= 14.041
  06:11:14 STEP 2249 H=2249:00 +CPU= 12.922
  06:11:36 STEP 2250 H=2250:00 +CPU= 20.996
  06:11:49 STEP 2251 H=2251:00 +CPU= 12.721
  06:12:03 STEP 2252 H=2252:00 +CPU= 13.344
  06:12:23 STEP 2253 H=2253:00 +CPU= 20.298
  06:12:35 STEP 2254 H=2254:00 +CPU= 11.866
  06:12:47 STEP 2255 H=2255:00 +CPU= 11.679
  06:13:09 STEP 2256 H=2256:00 +CPU= 21.577
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296_93.zip
Uploading the intermediate file: upload_file_93.zip
  06:13:21 STEP 2257 H=2257:00 +CPU= 11.558
  06:13:36 STEP 2258 H=2258:00 +CPU= 14.371
  06:13:58 STEP 2259 H=2259:00 +CPU= 21.726
  06:14:13 STEP 2260 H=2260:00 +CPU= 14.030
  06:14:26 STEP 2261 H=2261:00 +CPU= 13.476
  06:14:49 STEP 2262 H=2262:00 +CPU= 22.062
  06:15:03 STEP 2263 H=2263:00 +CPU= 13.225
  06:15:16 STEP 2264 H=2264:00 +CPU= 12.735
  06:15:36 STEP 2265 H=2265:00 +CPU= 19.457
  06:15:49 STEP 2266 H=2266:00 +CPU= 12.505
  06:16:01 STEP 2267 H=2267:00 +CPU= 12.154
  06:16:26 STEP 2268 H=2268:00 +CPU= 24.304
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002268
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0610
(argv4) batchid: 983
(argv5) wuid: 12200254
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/22
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12200254
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/22/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/22/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ca16c99ed4fc05b622ad3edd54bfd939 to: /var/lib/boinc/slots/22/oifs_43r3_ps_0610_2014050100_123_983_12200254.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/22/oifs_43r3_ps_0610_2014050100_123_983_12200254.zip
ic_ancil_file: ic_ancil_12200254
ifsdata_file: ifsdata_12200254
climate_data_file: clim_data_12200254
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4fa1867a339f43ddea47b8a0939bcf72 to: /var/lib/boinc/slots/22/ic_ancil_12200254.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/22/ic_ancil_12200254.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_19bf4f0ca2f9cb95c91631f4d49697c7 to: /var/lib/boinc/slots/22/ifsdata/ifsdata_12200254.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/22/ifsdata/ifsdata_12200254.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/22/159l_2/clim_data_12200254.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/22/159l_2/clim_data_12200254.zip
Checking for progress XML file: /var/lib/boinc/slots/22/progress_file_12200254.xml
Opened progress file ok : /var/lib/boinc/slots/22/progress_file_12200254.xml
-- Model is restarting --
Adjusting last_iter, 2269, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/22/progress_file_12200254.xml
last_cpu_time: 43859
upload_file_number: 94
last_iter: 2255
last_upload: 8121600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0610_2014050100_123_983_12200254_1_r1352058296
The child process has been launched with process id: 14301
Executing the command: /var/lib/boinc/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:14301:14301] [20230125:061906:1674623946:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:19:39 STEP 2256 H=2256:00 +CPU= 27.318
  06:19:54 STEP 2257 H=2257:00 +CPU= 15.155
  06:20:10 STEP 2258 H=2258:00 +CPU= 15.033
  06:20:34 STEP 2259 H=2259:00 +CPU= 23.275
  06:20:48 STEP 2260 H=2260:00 +CPU= 13.345
  06:21:03 STEP 2261 H=2261:00 +CPU= 14.424
  06:21:24 STEP 2262 H=2262:00 +CPU= 20.801
  06:21:37 STEP 2263 H=2263:00 +CPU= 12.071
  06:21:49 STEP 2264 H=2264:00 +CPU= 12.028
  06:22:08 STEP 2265 H=2265:00 +CPU= 18.540
  06:22:19 STEP 2266 H=2266:00 +CPU= 11.601
  06:22:31 STEP 2267 H=2267:00 +CPU= 11.016
  06:22:51 STEP 2268 H=2268:00 +CPU= 19.857
Moving to projects directory: /var/lib/boinc/slots/22/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/22/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/22/ICMUAhq0f+002268
  06:23:02 STEP 2269 H=2269:00 +CPU= 11.231
  06:23:15 STEP 2270 H=2270:00 +CPU= 12.509
  06:23:39 STEP 2271 H=2271:00 +CPU= 23.293
  06:23:54 STEP 2272 H=2272:00 +CPU= 14.355
  06:24:07 STEP 2273 H=2273:00 +CPU= 13.186
  06:24:29 STEP 2274 H=2274:00 +CPU= 21.365
  06:24:42 STEP 2275 H=2275:00 +CPU= 12.833
  06:24:56 STEP 2276 H=2276:00 +CPU= 13.219
  06:25:17 STEP 2277 H=2277:00 +CPU= 20.610
  06:25:30 STEP 2278 H=2278:00 +CPU= 13.210
  06:25:43 STEP 2279 H=2279:00 +CPU= 12.976
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:38:28 1499708 22302317 oifs_43r3_ps_0610_2014050100_123_983_12200254_1 5,875,200 31,821 0.0054
09 May 2023 16:38:24 1499708 22302317 oifs_43r3_ps_0610_2014050100_123_983_12200254_1 5,011,200 27,176 0.0054
09 May 2023 16:38:18 1499708 22302317 oifs_43r3_ps_0610_2014050100_123_983_12200254_1 4,147,200 22,536 0.0054
09 May 2023 16:38:12 1499708 22302317 oifs_43r3_ps_0610_2014050100_123_983_12200254_1 3,283,200 17,902 0.0055
09 May 2023 16:38:06 1499708 22302317 oifs_43r3_ps_0610_2014050100_123_983_12200254_1 2,419,200 13,277 0.0055
09 May 2023 16:37:44 1499708 22302317 oifs_43r3_ps_0610_2014050100_123_983_12200254_1 1,123,200 5,985 0.0053


©2024 cpdn.org