climateprediction.net home page
Task 22299999

Task 22299999

Name oifs_43r3_ps_0505_1981050100_123_950_12167149_1
Workunit 12167149
Created 19 Jan 2023, 19:44:44 UTC
Sent 19 Jan 2023, 19:51:20 UTC
Report deadline 20 Mar 2023, 19:51:20 UTC
Received 20 Jan 2023, 8:17:34 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1537625
Run time 12 hours 17 min 12 sec
CPU time 12 hours 17 min 12 sec
Validate state Invalid
Credit 12,096.00
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,779.03 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,019.05 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
12167149/ICMUAhq0f+001872
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001884
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001884
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001884
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_78.zip
Uploading the intermediate file: upload_file_78.zip
  16:52:11 STEP 1897 H=1897:00 +CPU= 10.569
  16:52:22 STEP 1898 H=1898:00 +CPU= 10.385
  16:52:38 STEP 1899 H=1899:00 +CPU= 16.426
  16:52:49 STEP 1900 H=1900:00 +CPU= 10.525
  16:52:59 STEP 1901 H=1901:00 +CPU= 10.126
  16:53:16 STEP 1902 H=1902:00 +CPU= 16.778
  16:53:26 STEP 1903 H=1903:00 +CPU= 10.512
  16:53:37 STEP 1904 H=1904:00 +CPU= 10.536
  16:53:53 STEP 1905 H=1905:00 +CPU= 16.495
  16:54:04 STEP 1906 H=1906:00 +CPU= 10.827
  16:54:14 STEP 1907 H=1907:00 +CPU= 10.377
  16:54:33 STEP 1908 H=1908:00 +CPU= 18.048
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001908
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001908
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001908
  16:54:43 STEP 1909 H=1909:00 +CPU= 10.312
  16:54:54 STEP 1910 H=1910:00 +CPU= 10.708
  16:55:10 STEP 1911 H=1911:00 +CPU= 16.657
  16:55:21 STEP 1912 H=1912:00 +CPU= 10.315
  16:55:31 STEP 1913 H=1913:00 +CPU= 10.359
  16:55:48 STEP 1914 H=1914:00 +CPU= 16.843
  16:55:58 STEP 1915 H=1915:00 +CPU= 10.161
  16:56:09 STEP 1916 H=1916:00 +CPU= 10.709
  16:56:25 STEP 1917 H=1917:00 +CPU= 16.313
  16:56:36 STEP 1918 H=1918:00 +CPU= 10.650
  16:56:46 STEP 1919 H=1919:00 +CPU= 10.440
  16:57:05 STEP 1920 H=1920:00 +CPU= 19.093
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001920
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001920
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001920
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001896
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001896
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001896
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001908
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001908
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001908
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_79.zip
Uploading the intermediate file: upload_file_79.zip
  16:57:15 STEP 1921 H=1921:00 +CPU= 10.316
  16:57:26 STEP 1922 H=1922:00 +CPU= 10.515
Uploading trickle at timestep: 6912000
  16:57:42 STEP 1923 H=1923:00 +CPU= 16.340
  16:57:53 STEP 1924 H=1924:00 +CPU= 10.400
  16:58:03 STEP 1925 H=1925:00 +CPU= 10.776
  16:58:20 STEP 1926 H=1926:00 +CPU= 16.947
  16:58:31 STEP 1927 H=1927:00 +CPU= 10.250
  16:58:41 STEP 1928 H=1928:00 +CPU= 10.727
  16:58:58 STEP 1929 H=1929:00 +CPU= 16.384
  16:59:09 STEP 1930 H=1930:00 +CPU= 10.862
  16:59:19 STEP 1931 H=1931:00 +CPU= 10.353
  16:59:37 STEP 1932 H=1932:00 +CPU= 18.225
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001932
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001932
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001932
  16:59:48 STEP 1933 H=1933:00 +CPU= 10.541
  16:59:58 STEP 1934 H=1934:00 +CPU= 10.464
  17:00:15 STEP 1935 H=1935:00 +CPU= 16.727
  17:00:25 STEP 1936 H=1936:00 +CPU= 10.452
  17:00:36 STEP 1937 H=1937:00 +CPU= 10.495
  17:00:53 STEP 1938 H=1938:00 +CPU= 17.266
  17:01:04 STEP 1939 H=1939:00 +CPU= 10.318
  17:01:14 STEP 1940 H=1940:00 +CPU= 10.610
  17:01:31 STEP 1941 H=1941:00 +CPU= 16.374
  17:01:42 STEP 1942 H=1942:00 +CPU= 11.039
  17:01:52 STEP 1943 H=1943:00 +CPU= 10.456
  17:02:12 STEP 1944 H=1944:00 +CPU= 19.245
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001944
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001944
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001944
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001920
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001920
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001920
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001932
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001932
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001932
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_80.zip
Uploading the intermediate file: upload_file_80.zip
  17:02:22 STEP 1945 H=1945:00 +CPU= 10.536
  17:02:33 STEP 1946 H=1946:00 +CPU= 10.517
  17:02:49 STEP 1947 H=1947:00 +CPU= 16.870
  17:03:00 STEP 1948 H=1948:00 +CPU= 10.444
  17:03:10 STEP 1949 H=1949:00 +CPU= 10.370
  17:03:27 STEP 1950 H=1950:00 +CPU= 16.875
  17:03:37 STEP 1951 H=1951:00 +CPU= 10.261
  17:03:48 STEP 1952 H=1952:00 +CPU= 10.549
  17:04:04 STEP 1953 H=1953:00 +CPU= 16.387
  17:04:15 STEP 1954 H=1954:00 +CPU= 10.299
  17:04:25 STEP 1955 H=1955:00 +CPU= 10.388
  17:04:43 STEP 1956 H=1956:00 +CPU= 17.914
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001956
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001956
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001956
  17:04:53 STEP 1957 H=1957:00 +CPU= 10.184
  17:05:04 STEP 1958 H=1958:00 +CPU= 10.701
  17:05:20 STEP 1959 H=1959:00 +CPU= 16.198
  17:05:31 STEP 1960 H=1960:00 +CPU= 10.476
  17:05:41 STEP 1961 H=1961:00 +CPU= 10.121
  17:05:58 STEP 1962 H=1962:00 +CPU= 16.942
  17:06:08 STEP 1963 H=1963:00 +CPU= 10.182
  17:06:18 STEP 1964 H=1964:00 +CPU= 10.326
  17:06:34 STEP 1965 H=1965:00 +CPU= 16.309
  17:06:45 STEP 1966 H=1966:00 +CPU= 10.474
  17:06:56 STEP 1967 H=1967:00 +CPU= 10.587
  17:07:15 STEP 1968 H=1968:00 +CPU= 19.080
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001968
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001968
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001968
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001944
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001944
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001944
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001956
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001956
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001956
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_81.zip
Uploading the intermediate file: upload_file_81.zip
  17:07:25 STEP 1969 H=1969:00 +CPU= 10.303
  17:07:36 STEP 1970 H=1970:00 +CPU= 10.516
  17:07:52 STEP 1971 H=1971:00 +CPU= 16.280
  17:08:03 STEP 1972 H=1972:00 +CPU= 10.694
  17:08:13 STEP 1973 H=1973:00 +CPU= 10.427
  17:08:29 STEP 1974 H=1974:00 +CPU= 16.220
  17:08:40 STEP 1975 H=1975:00 +CPU= 10.553
  17:08:50 STEP 1976 H=1976:00 +CPU= 10.405
  17:09:07 STEP 1977 H=1977:00 +CPU= 16.407
  17:09:17 STEP 1978 H=1978:00 +CPU= 10.259
  17:09:27 STEP 1979 H=1979:00 +CPU= 10.101
  17:09:45 STEP 1980 H=1980:00 +CPU= 17.908
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001980
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001980
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001980
  17:09:55 STEP 1981 H=1981:00 +CPU= 10.087
  17:10:06 STEP 1982 H=1982:00 +CPU= 10.598
  17:10:22 STEP 1983 H=1983:00 +CPU= 16.053
  17:10:32 STEP 1984 H=1984:00 +CPU= 10.274
  17:10:42 STEP 1985 H=1985:00 +CPU= 10.136
  17:10:59 STEP 1986 H=1986:00 +CPU= 16.650
  17:11:09 STEP 1987 H=1987:00 +CPU= 10.359
  17:11:20 STEP 1988 H=1988:00 +CPU= 10.377
  17:11:36 STEP 1989 H=1989:00 +CPU= 16.613
  17:11:47 STEP 1990 H=1990:00 +CPU= 10.658
  17:11:57 STEP 1991 H=1991:00 +CPU= 10.340
  17:12:18 STEP 1992 H=1992:00 +CPU= 19.124
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+001992
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+001992
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+001992
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001968
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001968
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001968
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001980
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001980
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001980
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_82.zip
Uploading the intermediate file: upload_file_82.zip
  17:12:28 STEP 1993 H=1993:00 +CPU= 10.093
  17:12:39 STEP 1994 H=1994:00 +CPU= 10.554
  17:12:55 STEP 1995 H=1995:00 +CPU= 16.491
  17:13:06 STEP 1996 H=1996:00 +CPU= 10.898
  17:13:16 STEP 1997 H=1997:00 +CPU= 10.118
  17:13:33 STEP 1998 H=1998:00 +CPU= 16.764
  17:13:43 STEP 1999 H=1999:00 +CPU= 10.112
  17:13:54 STEP 2000 H=2000:00 +CPU= 10.542
  17:14:11 STEP 2001 H=2001:00 +CPU= 16.684
  17:14:21 STEP 2002 H=2002:00 +CPU= 10.471
  17:14:31 STEP 2003 H=2003:00 +CPU= 10.164
  17:14:50 STEP 2004 H=2004:00 +CPU= 18.536
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002004
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002004
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002004
  17:15:00 STEP 2005 H=2005:00 +CPU= 10.146
  17:15:10 STEP 2006 H=2006:00 +CPU= 10.237
  17:15:27 STEP 2007 H=2007:00 +CPU= 16.847
  17:15:37 STEP 2008 H=2008:00 +CPU= 10.396
  17:15:48 STEP 2009 H=2009:00 +CPU= 10.337
  17:16:04 STEP 2010 H=2010:00 +CPU= 16.263
  17:16:14 STEP 2011 H=2011:00 +CPU= 10.345
  17:16:25 STEP 2012 H=2012:00 +CPU= 10.212
  17:16:41 STEP 2013 H=2013:00 +CPU= 16.586
  17:16:51 STEP 2014 H=2014:00 +CPU= 10.252
  17:17:02 STEP 2015 H=2015:00 +CPU= 10.472
  17:17:21 STEP 2016 H=2016:00 +CPU= 19.236
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002016
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002016
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002016
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+001992
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+001992
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+001992
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002004
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002004
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002004
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_83.zip
  17:17:32 STEP 2017 H=2017:00 +CPU= 10.223
Uploading the intermediate file: upload_file_83.zip
  17:17:42 STEP 2018 H=2018:00 +CPU= 10.572
  17:17:59 STEP 2019 H=2019:00 +CPU= 16.335
  17:18:09 STEP 2020 H=2020:00 +CPU= 10.336
  17:18:19 STEP 2021 H=2021:00 +CPU= 10.211
  17:18:36 STEP 2022 H=2022:00 +CPU= 16.542
  17:18:46 STEP 2023 H=2023:00 +CPU= 10.315
  17:18:56 STEP 2024 H=2024:00 +CPU= 10.283
  17:19:13 STEP 2025 H=2025:00 +CPU= 16.557
  17:19:23 STEP 2026 H=2026:00 +CPU= 10.468
  17:19:33 STEP 2027 H=2027:00 +CPU= 10.076
  17:19:51 STEP 2028 H=2028:00 +CPU= 18.105
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002028
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002028
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002028
  17:20:02 STEP 2029 H=2029:00 +CPU= 10.280
  17:20:12 STEP 2030 H=2030:00 +CPU= 10.357
  17:20:28 STEP 2031 H=2031:00 +CPU= 16.100
  17:20:39 STEP 2032 H=2032:00 +CPU= 10.377
  17:20:49 STEP 2033 H=2033:00 +CPU= 10.044
  17:21:05 STEP 2034 H=2034:00 +CPU= 16.417
  17:21:15 STEP 2035 H=2035:00 +CPU= 10.280
  17:21:25 STEP 2036 H=2036:00 +CPU= 10.073
  17:21:42 STEP 2037 H=2037:00 +CPU= 16.884
  17:21:53 STEP 2038 H=2038:00 +CPU= 10.526
  17:22:03 STEP 2039 H=2039:00 +CPU= 10.302
  17:22:22 STEP 2040 H=2040:00 +CPU= 18.707
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002040
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002040
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002040
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002016
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002016
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002016
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002028
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002028
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002028
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_84.zip
  17:22:33 STEP 2041 H=2041:00 +CPU= 10.378
Uploading the intermediate file: upload_file_84.zip
  17:22:43 STEP 2042 H=2042:00 +CPU= 10.285
  17:22:59 STEP 2043 H=2043:00 +CPU= 16.411
  17:23:10 STEP 2044 H=2044:00 +CPU= 10.556
  17:23:20 STEP 2045 H=2045:00 +CPU= 10.122
  17:23:36 STEP 2046 H=2046:00 +CPU= 16.437
  17:23:46 STEP 2047 H=2047:00 +CPU= 10.021
  17:23:57 STEP 2048 H=2048:00 +CPU= 10.634
  17:24:13 STEP 2049 H=2049:00 +CPU= 16.039
  17:24:23 STEP 2050 H=2050:00 +CPU= 10.333
  17:24:34 STEP 2051 H=2051:00 +CPU= 10.085
  17:24:51 STEP 2052 H=2052:00 +CPU= 17.839
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002052
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002052
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002052
  17:25:02 STEP 2053 H=2053:00 +CPU= 10.254
  17:25:12 STEP 2054 H=2054:00 +CPU= 10.282
  17:25:28 STEP 2055 H=2055:00 +CPU= 16.350
  17:25:39 STEP 2056 H=2056:00 +CPU= 10.429
  17:25:49 STEP 2057 H=2057:00 +CPU=  9.934
  17:26:05 STEP 2058 H=2058:00 +CPU= 16.559
  17:26:15 STEP 2059 H=2059:00 +CPU= 10.163
  17:26:26 STEP 2060 H=2060:00 +CPU= 10.203
  17:26:42 STEP 2061 H=2061:00 +CPU= 16.270
  17:26:52 STEP 2062 H=2062:00 +CPU= 10.566
  17:27:03 STEP 2063 H=2063:00 +CPU= 11.001
  17:27:25 STEP 2064 H=2064:00 +CPU= 19.053
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002064
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002064
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002064
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002040
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002040
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002040
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002052
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002052
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002052
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_85.zip
Uploading the intermediate file: upload_file_85.zip
  17:27:35 STEP 2065 H=2065:00 +CPU= 10.505
  17:27:45 STEP 2066 H=2066:00 +CPU= 10.397
  17:28:02 STEP 2067 H=2067:00 +CPU= 16.298
  17:28:12 STEP 2068 H=2068:00 +CPU= 10.443
  17:28:22 STEP 2069 H=2069:00 +CPU= 10.293
  17:28:39 STEP 2070 H=2070:00 +CPU= 16.533
  17:28:49 STEP 2071 H=2071:00 +CPU= 10.220
  17:29:00 STEP 2072 H=2072:00 +CPU= 10.437
  17:29:16 STEP 2073 H=2073:00 +CPU= 16.516
  17:29:27 STEP 2074 H=2074:00 +CPU= 10.798
  17:29:37 STEP 2075 H=2075:00 +CPU= 10.207
  17:29:55 STEP 2076 H=2076:00 +CPU= 17.749
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002076
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002076
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002076
  17:30:05 STEP 2077 H=2077:00 +CPU=  9.947
  17:30:16 STEP 2078 H=2078:00 +CPU= 10.808
  17:30:32 STEP 2079 H=2079:00 +CPU= 16.078
  17:30:42 STEP 2080 H=2080:00 +CPU= 10.567
  17:30:52 STEP 2081 H=2081:00 +CPU= 10.104
  17:31:09 STEP 2082 H=2082:00 +CPU= 16.844
  17:31:20 STEP 2083 H=2083:00 +CPU= 10.438
  17:31:30 STEP 2084 H=2084:00 +CPU= 10.415
  17:31:47 STEP 2085 H=2085:00 +CPU= 16.510
  17:31:57 STEP 2086 H=2086:00 +CPU= 10.494
  17:32:08 STEP 2087 H=2087:00 +CPU= 10.659
  17:32:30 STEP 2088 H=2088:00 +CPU= 19.581
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002088
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002088
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002088
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002064
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002064
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002064
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002076
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002076
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002076
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_86.zip
Uploading the intermediate file: upload_file_86.zip
  17:32:40 STEP 2089 H=2089:00 +CPU=  9.997
  17:32:51 STEP 2090 H=2090:00 +CPU= 10.817
  17:33:07 STEP 2091 H=2091:00 +CPU= 16.306
  17:33:17 STEP 2092 H=2092:00 +CPU= 10.451
  17:33:28 STEP 2093 H=2093:00 +CPU= 10.462
  17:33:44 STEP 2094 H=2094:00 +CPU= 16.631
  17:33:54 STEP 2095 H=2095:00 +CPU= 10.065
  17:34:05 STEP 2096 H=2096:00 +CPU= 10.210
  17:34:21 STEP 2097 H=2097:00 +CPU= 16.589
  17:34:32 STEP 2098 H=2098:00 +CPU= 10.403
  17:34:42 STEP 2099 H=2099:00 +CPU= 10.138
  17:35:00 STEP 2100 H=2100:00 +CPU= 18.014
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002100
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002100
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002100
  17:35:10 STEP 2101 H=2101:00 +CPU= 10.132
  17:35:21 STEP 2102 H=2102:00 +CPU= 10.651
  17:35:37 STEP 2103 H=2103:00 +CPU= 16.304
  17:35:48 STEP 2104 H=2104:00 +CPU= 10.855
  17:35:58 STEP 2105 H=2105:00 +CPU= 10.084
  17:36:14 STEP 2106 H=2106:00 +CPU= 16.337
  17:36:24 STEP 2107 H=2107:00 +CPU= 10.063
  17:36:35 STEP 2108 H=2108:00 +CPU= 10.243
  17:36:51 STEP 2109 H=2109:00 +CPU= 16.623
  17:37:02 STEP 2110 H=2110:00 +CPU= 10.397
  17:37:12 STEP 2111 H=2111:00 +CPU= 10.252
  17:37:31 STEP 2112 H=2112:00 +CPU= 18.969
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002112
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002112
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002112
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002088
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002088
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002088
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002100
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002100
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002100
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_87.zip
Uploading the intermediate file: upload_file_87.zip
  17:37:41 STEP 2113 H=2113:00 +CPU= 10.153
  17:37:52 STEP 2114 H=2114:00 +CPU= 10.679
  17:38:08 STEP 2115 H=2115:00 +CPU= 16.062
  17:38:19 STEP 2116 H=2116:00 +CPU= 10.841
  17:38:29 STEP 2117 H=2117:00 +CPU=  9.996
  17:38:45 STEP 2118 H=2118:00 +CPU= 16.707
  17:38:55 STEP 2119 H=2119:00 +CPU= 10.233
  17:39:06 STEP 2120 H=2120:00 +CPU= 10.412
  17:39:22 STEP 2121 H=2121:00 +CPU= 16.444
  17:39:33 STEP 2122 H=2122:00 +CPU= 10.371
  17:39:43 STEP 2123 H=2123:00 +CPU= 10.033
  17:40:01 STEP 2124 H=2124:00 +CPU= 18.258
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002124
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002124
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002124
  17:40:11 STEP 2125 H=2125:00 +CPU= 10.013
  17:40:22 STEP 2126 H=2126:00 +CPU= 10.607
  17:40:38 STEP 2127 H=2127:00 +CPU= 16.690
  17:40:49 STEP 2128 H=2128:00 +CPU= 10.649
  17:40:59 STEP 2129 H=2129:00 +CPU= 10.307
  17:41:16 STEP 2130 H=2130:00 +CPU= 16.393
  17:41:26 STEP 2131 H=2131:00 +CPU= 10.198
  17:41:36 STEP 2132 H=2132:00 +CPU= 10.287
  17:41:52 STEP 2133 H=2133:00 +CPU= 16.123
  17:42:03 STEP 2134 H=2134:00 +CPU= 10.405
  17:42:13 STEP 2135 H=2135:00 +CPU= 10.392
  17:42:32 STEP 2136 H=2136:00 +CPU= 19.348
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002136
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002136
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002136
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002112
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002112
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002112
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002124
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002124
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002124
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_88.zip
Uploading the intermediate file: upload_file_88.zip
  17:42:43 STEP 2137 H=2137:00 +CPU= 10.069
  17:42:53 STEP 2138 H=2138:00 +CPU= 10.754
  17:43:10 STEP 2139 H=2139:00 +CPU= 16.782
  17:43:20 STEP 2140 H=2140:00 +CPU= 10.403
  17:43:31 STEP 2141 H=2141:00 +CPU= 10.064
  17:43:47 STEP 2142 H=2142:00 +CPU= 16.430
  17:43:57 STEP 2143 H=2143:00 +CPU= 10.220
  17:44:08 STEP 2144 H=2144:00 +CPU= 10.504
  17:44:24 STEP 2145 H=2145:00 +CPU= 16.153
  17:44:35 STEP 2146 H=2146:00 +CPU= 10.763
  17:44:45 STEP 2147 H=2147:00 +CPU= 10.141
  17:45:03 STEP 2148 H=2148:00 +CPU= 18.136
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002148
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002148
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002148
  17:45:13 STEP 2149 H=2149:00 +CPU= 10.170
  17:45:23 STEP 2150 H=2150:00 +CPU= 10.291
  17:45:40 STEP 2151 H=2151:00 +CPU= 16.535
  17:45:50 STEP 2152 H=2152:00 +CPU= 10.501
  17:46:01 STEP 2153 H=2153:00 +CPU= 10.124
  17:46:17 STEP 2154 H=2154:00 +CPU= 16.869
  17:46:28 STEP 2155 H=2155:00 +CPU= 10.165
  17:46:38 STEP 2156 H=2156:00 +CPU= 10.180
  17:46:54 STEP 2157 H=2157:00 +CPU= 16.484
  17:47:05 STEP 2158 H=2158:00 +CPU= 10.376
  17:47:15 STEP 2159 H=2159:00 +CPU= 10.557
  17:47:35 STEP 2160 H=2160:00 +CPU= 19.118
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002160
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002160
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002160
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002136
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002136
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002136
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002148
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002148
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002148
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_89.zip
Uploading the intermediate file: upload_file_89.zip
  17:47:45 STEP 2161 H=2161:00 +CPU= 10.337
  17:47:56 STEP 2162 H=2162:00 +CPU= 10.612
Uploading trickle at timestep: 7776000
  17:48:12 STEP 2163 H=2163:00 +CPU= 16.037
  17:48:22 STEP 2164 H=2164:00 +CPU= 10.549
  17:48:32 STEP 2165 H=2165:00 +CPU= 10.266
  17:48:49 STEP 2166 H=2166:00 +CPU= 16.620
  17:48:59 STEP 2167 H=2167:00 +CPU= 10.132
  17:49:10 STEP 2168 H=2168:00 +CPU= 10.398
  17:49:26 STEP 2169 H=2169:00 +CPU= 16.392
  17:49:36 STEP 2170 H=2170:00 +CPU= 10.398
  17:49:47 STEP 2171 H=2171:00 +CPU= 10.540
  17:50:05 STEP 2172 H=2172:00 +CPU= 17.980
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002172
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002172
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002172
  17:50:15 STEP 2173 H=2173:00 +CPU= 10.351
  17:50:26 STEP 2174 H=2174:00 +CPU= 10.515
  17:50:42 STEP 2175 H=2175:00 +CPU= 16.176
  17:50:52 STEP 2176 H=2176:00 +CPU= 10.458
  17:51:03 STEP 2177 H=2177:00 +CPU= 10.177
  17:51:19 STEP 2178 H=2178:00 +CPU= 16.369
  17:51:29 STEP 2179 H=2179:00 +CPU= 10.282
  17:51:40 STEP 2180 H=2180:00 +CPU= 10.409
  17:51:56 STEP 2181 H=2181:00 +CPU= 16.759
  17:52:07 STEP 2182 H=2182:00 +CPU= 10.231
  17:52:17 STEP 2183 H=2183:00 +CPU= 10.423
  17:52:38 STEP 2184 H=2184:00 +CPU= 19.132
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002184
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002184
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002184
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002160
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002160
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002160
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002172
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002172
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002172
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_90.zip
Uploading the intermediate file: upload_file_90.zip
  17:52:48 STEP 2185 H=2185:00 +CPU= 10.004
  17:52:58 STEP 2186 H=2186:00 +CPU= 10.483
  17:53:14 STEP 2187 H=2187:00 +CPU= 16.408
  17:53:25 STEP 2188 H=2188:00 +CPU= 10.452
  17:53:35 STEP 2189 H=2189:00 +CPU= 10.001
  17:53:51 STEP 2190 H=2190:00 +CPU= 16.607
  17:54:02 STEP 2191 H=2191:00 +CPU= 10.394
  17:54:12 STEP 2192 H=2192:00 +CPU= 10.273
  17:54:29 STEP 2193 H=2193:00 +CPU= 16.724
  17:54:39 STEP 2194 H=2194:00 +CPU= 10.269
  17:54:49 STEP 2195 H=2195:00 +CPU= 10.069
  17:55:07 STEP 2196 H=2196:00 +CPU= 17.826
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002196
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002196
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002196
  17:55:17 STEP 2197 H=2197:00 +CPU= 10.091
  17:55:28 STEP 2198 H=2198:00 +CPU= 10.692
  17:55:44 STEP 2199 H=2199:00 +CPU= 15.970
  17:55:54 STEP 2200 H=2200:00 +CPU= 10.339
  17:56:04 STEP 2201 H=2201:00 +CPU= 10.223
  17:56:21 STEP 2202 H=2202:00 +CPU= 16.365
  17:56:31 STEP 2203 H=2203:00 +CPU=  9.903
  17:56:41 STEP 2204 H=2204:00 +CPU= 10.431
  17:56:57 STEP 2205 H=2205:00 +CPU= 16.200
  17:57:08 STEP 2206 H=2206:00 +CPU= 10.589
  17:57:18 STEP 2207 H=2207:00 +CPU= 10.359
  17:57:39 STEP 2208 H=2208:00 +CPU= 19.144
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002208
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002208
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002208
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002184
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002184
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002184
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002196
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002196
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002196
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_91.zip
Uploading the intermediate file: upload_file_91.zip
  17:57:49 STEP 2209 H=2209:00 +CPU= 10.236
  17:57:59 STEP 2210 H=2210:00 +CPU= 10.453
  17:58:16 STEP 2211 H=2211:00 +CPU= 16.904
  17:58:27 STEP 2212 H=2212:00 +CPU= 10.454
  17:58:37 STEP 2213 H=2213:00 +CPU= 10.397
  17:58:54 STEP 2214 H=2214:00 +CPU= 16.569
  17:59:04 STEP 2215 H=2215:00 +CPU= 10.451
  17:59:15 STEP 2216 H=2216:00 +CPU= 10.654
  17:59:31 STEP 2217 H=2217:00 +CPU= 16.607
  17:59:42 STEP 2218 H=2218:00 +CPU= 10.353
  17:59:52 STEP 2219 H=2219:00 +CPU= 10.190
  18:00:10 STEP 2220 H=2220:00 +CPU= 18.216
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002220
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002220
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002220
  18:00:20 STEP 2221 H=2221:00 +CPU= 10.300
  18:00:31 STEP 2222 H=2222:00 +CPU= 10.278
  18:00:47 STEP 2223 H=2223:00 +CPU= 16.163
  18:00:57 STEP 2224 H=2224:00 +CPU= 10.571
  18:01:07 STEP 2225 H=2225:00 +CPU= 10.107
  18:01:24 STEP 2226 H=2226:00 +CPU= 16.990
  18:01:35 STEP 2227 H=2227:00 +CPU= 10.227
  18:01:45 STEP 2228 H=2228:00 +CPU= 10.334
  18:02:01 STEP 2229 H=2229:00 +CPU= 16.285
  18:02:12 STEP 2230 H=2230:00 +CPU= 10.738
  18:02:23 STEP 2231 H=2231:00 +CPU= 10.472
  18:02:41 STEP 2232 H=2232:00 +CPU= 18.819
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002232
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002232
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002232
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002208
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002208
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002208
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002220
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002220
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002220
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_92.zip
Uploading the intermediate file: upload_file_92.zip
  18:02:52 STEP 2233 H=2233:00 +CPU= 10.148
  18:03:02 STEP 2234 H=2234:00 +CPU= 10.465
  18:03:18 STEP 2235 H=2235:00 +CPU= 16.398
  18:03:29 STEP 2236 H=2236:00 +CPU= 10.298
  18:03:39 STEP 2237 H=2237:00 +CPU= 10.020
  18:03:56 STEP 2238 H=2238:00 +CPU= 16.899
  18:04:06 STEP 2239 H=2239:00 +CPU=  9.926
  18:04:16 STEP 2240 H=2240:00 +CPU= 10.469
  18:04:32 STEP 2241 H=2241:00 +CPU= 16.201
  18:04:43 STEP 2242 H=2242:00 +CPU= 10.435
  18:04:53 STEP 2243 H=2243:00 +CPU=  9.915
  18:05:11 STEP 2244 H=2244:00 +CPU= 18.337
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002244
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002244
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002244
  18:05:21 STEP 2245 H=2245:00 +CPU= 10.195
  18:05:31 STEP 2246 H=2246:00 +CPU= 10.389
  18:05:48 STEP 2247 H=2247:00 +CPU= 16.142
  18:05:58 STEP 2248 H=2248:00 +CPU= 10.370
  18:06:08 STEP 2249 H=2249:00 +CPU= 10.183
  18:06:25 STEP 2250 H=2250:00 +CPU= 16.538
  18:06:35 STEP 2251 H=2251:00 +CPU= 10.271
  18:06:45 STEP 2252 H=2252:00 +CPU= 10.160
  18:07:02 STEP 2253 H=2253:00 +CPU= 16.446
  18:07:12 STEP 2254 H=2254:00 +CPU= 10.367
  18:07:22 STEP 2255 H=2255:00 +CPU= 10.493
  18:07:43 STEP 2256 H=2256:00 +CPU= 18.783
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002256
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002256
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002256
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002232
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002232
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002232
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002244
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002244
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002244
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_93.zip
Uploading the intermediate file: upload_file_93.zip
  18:07:54 STEP 2257 H=2257:00 +CPU= 10.600
  18:08:04 STEP 2258 H=2258:00 +CPU= 10.475
  18:08:21 STEP 2259 H=2259:00 +CPU= 16.354
  18:08:31 STEP 2260 H=2260:00 +CPU= 10.363
  18:08:41 STEP 2261 H=2261:00 +CPU= 10.174
  18:08:58 STEP 2262 H=2262:00 +CPU= 16.901
  18:09:08 STEP 2263 H=2263:00 +CPU= 10.075
  18:09:18 STEP 2264 H=2264:00 +CPU= 10.248
  18:09:35 STEP 2265 H=2265:00 +CPU= 16.458
  18:09:45 STEP 2266 H=2266:00 +CPU= 10.369
  18:09:55 STEP 2267 H=2267:00 +CPU= 10.061
  18:10:14 STEP 2268 H=2268:00 +CPU= 18.885
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002268
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002268
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002268
  18:10:25 STEP 2269 H=2269:00 +CPU= 10.355
  18:10:35 STEP 2270 H=2270:00 +CPU= 10.360
  18:10:51 STEP 2271 H=2271:00 +CPU= 16.365
  18:11:02 STEP 2272 H=2272:00 +CPU= 10.924
  18:11:13 STEP 2273 H=2273:00 +CPU= 10.243
  18:11:29 STEP 2274 H=2274:00 +CPU= 16.401
  18:11:39 STEP 2275 H=2275:00 +CPU= 10.158
  18:11:49 STEP 2276 H=2276:00 +CPU= 10.236
  18:12:06 STEP 2277 H=2277:00 +CPU= 16.210
  18:12:16 STEP 2278 H=2278:00 +CPU= 10.285
  18:12:26 STEP 2279 H=2279:00 +CPU= 10.554
  18:12:46 STEP 2280 H=2280:00 +CPU= 19.457
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002280
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002280
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002256
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002256
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002256
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_94.zip
Uploading the intermediate file: upload_file_94.zip
  18:12:56 STEP 2281 H=2281:00 +CPU= 10.345
  18:13:07 STEP 2282 H=2282:00 +CPU= 10.338
  18:13:23 STEP 2283 H=2283:00 +CPU= 16.142
  18:13:34 STEP 2284 H=2284:00 +CPU= 10.866
  18:13:44 STEP 2285 H=2285:00 +CPU= 10.499
  18:14:01 STEP 2286 H=2286:00 +CPU= 16.471
  18:14:11 STEP 2287 H=2287:00 +CPU= 10.160
  18:14:21 STEP 2288 H=2288:00 +CPU= 10.474
  18:14:38 STEP 2289 H=2289:00 +CPU= 16.687
  18:14:48 STEP 2290 H=2290:00 +CPU= 10.200
  18:14:59 STEP 2291 H=2291:00 +CPU= 10.199
  18:15:17 STEP 2292 H=2292:00 +CPU= 18.006
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002292
  18:15:27 STEP 2293 H=2293:00 +CPU= 10.154
  18:15:37 STEP 2294 H=2294:00 +CPU= 10.800
  18:15:54 STEP 2295 H=2295:00 +CPU= 16.060
  18:16:04 STEP 2296 H=2296:00 +CPU= 10.366
  18:16:14 STEP 2297 H=2297:00 +CPU= 10.116
  18:16:31 STEP 2298 H=2298:00 +CPU= 16.589
  18:16:41 STEP 2299 H=2299:00 +CPU= 10.597
  18:16:52 STEP 2300 H=2300:00 +CPU= 10.345
  18:17:08 STEP 2301 H=2301:00 +CPU= 16.427
  18:17:19 STEP 2302 H=2302:00 +CPU= 10.632
  18:17:29 STEP 2303 H=2303:00 +CPU= 10.643
  18:17:48 STEP 2304 H=2304:00 +CPU= 18.899
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_95.zip
Uploading the intermediate file: upload_file_95.zip
  18:17:59 STEP 2305 H=2305:00 +CPU= 10.183
  18:18:09 STEP 2306 H=2306:00 +CPU= 10.278
  18:18:26 STEP 2307 H=2307:00 +CPU= 16.680
  18:18:36 STEP 2308 H=2308:00 +CPU= 10.255
  18:18:46 STEP 2309 H=2309:00 +CPU= 10.380
  18:19:03 STEP 2310 H=2310:00 +CPU= 16.449
  18:19:13 STEP 2311 H=2311:00 +CPU= 10.006
  18:19:23 STEP 2312 H=2312:00 +CPU= 10.821
  18:19:40 STEP 2313 H=2313:00 +CPU= 16.289
  18:19:50 STEP 2314 H=2314:00 +CPU= 10.607
  18:20:00 STEP 2315 H=2315:00 +CPU= 10.138
  18:20:19 STEP 2316 H=2316:00 +CPU= 18.536
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002316
  18:20:29 STEP 2317 H=2317:00 +CPU= 10.103
  18:20:40 STEP 2318 H=2318:00 +CPU= 10.488
  18:20:56 STEP 2319 H=2319:00 +CPU= 16.289
  18:21:06 STEP 2320 H=2320:00 +CPU= 10.410
  18:21:16 STEP 2321 H=2321:00 +CPU= 10.057
  18:21:33 STEP 2322 H=2322:00 +CPU= 16.925
  18:21:43 STEP 2323 H=2323:00 +CPU= 10.110
  18:21:54 STEP 2324 H=2324:00 +CPU= 10.440
  18:22:10 STEP 2325 H=2325:00 +CPU= 15.914
  18:22:20 STEP 2326 H=2326:00 +CPU= 10.516
  18:22:31 STEP 2327 H=2327:00 +CPU= 10.497
  18:22:51 STEP 2328 H=2328:00 +CPU= 19.244
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_96.zip
Uploading the intermediate file: upload_file_96.zip
  18:23:01 STEP 2329 H=2329:00 +CPU= 10.130
  18:23:11 STEP 2330 H=2330:00 +CPU= 10.487
  18:23:28 STEP 2331 H=2331:00 +CPU= 16.337
  18:23:38 STEP 2332 H=2332:00 +CPU= 10.392
  18:23:48 STEP 2333 H=2333:00 +CPU= 10.011
  18:24:06 STEP 2334 H=2334:00 +CPU= 17.276
  18:24:16 STEP 2335 H=2335:00 +CPU= 10.075
  18:24:26 STEP 2336 H=2336:00 +CPU= 10.554
  18:24:42 STEP 2337 H=2337:00 +CPU= 16.265
  18:24:53 STEP 2338 H=2338:00 +CPU= 10.517
  18:25:03 STEP 2339 H=2339:00 +CPU= 10.192
  18:25:21 STEP 2340 H=2340:00 +CPU= 17.900
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002340
  18:25:31 STEP 2341 H=2341:00 +CPU= 10.460
  18:25:42 STEP 2342 H=2342:00 +CPU= 10.536
  18:25:58 STEP 2343 H=2343:00 +CPU= 16.205
  18:26:09 STEP 2344 H=2344:00 +CPU= 10.712
  18:26:19 STEP 2345 H=2345:00 +CPU= 10.095
  18:26:36 STEP 2346 H=2346:00 +CPU= 16.566
  18:26:46 STEP 2347 H=2347:00 +CPU= 10.110
  18:26:56 STEP 2348 H=2348:00 +CPU= 10.272
  18:27:13 STEP 2349 H=2349:00 +CPU= 16.640
  18:27:23 STEP 2350 H=2350:00 +CPU= 10.163
  18:27:34 STEP 2351 H=2351:00 +CPU= 10.860
  18:27:56 STEP 2352 H=2352:00 +CPU= 19.003
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_97.zip
Uploading the intermediate file: upload_file_97.zip
  18:28:07 STEP 2353 H=2353:00 +CPU= 10.337
  18:28:17 STEP 2354 H=2354:00 +CPU= 10.564
  18:28:34 STEP 2355 H=2355:00 +CPU= 16.771
  18:28:44 STEP 2356 H=2356:00 +CPU= 10.262
  18:28:55 STEP 2357 H=2357:00 +CPU= 10.274
  18:29:11 STEP 2358 H=2358:00 +CPU= 16.509
  18:29:21 STEP 2359 H=2359:00 +CPU= 10.209
  18:29:32 STEP 2360 H=2360:00 +CPU= 10.715
  18:29:48 STEP 2361 H=2361:00 +CPU= 16.299
  18:29:59 STEP 2362 H=2362:00 +CPU= 10.510
  18:30:09 STEP 2363 H=2363:00 +CPU= 10.369
  18:30:27 STEP 2364 H=2364:00 +CPU= 17.909
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002364
  18:30:37 STEP 2365 H=2365:00 +CPU= 10.269
  18:30:48 STEP 2366 H=2366:00 +CPU= 10.568
  18:31:04 STEP 2367 H=2367:00 +CPU= 16.199
  18:31:14 STEP 2368 H=2368:00 +CPU= 10.259
  18:31:24 STEP 2369 H=2369:00 +CPU= 10.063
  18:31:41 STEP 2370 H=2370:00 +CPU= 16.673
  18:31:51 STEP 2371 H=2371:00 +CPU= 10.127
  18:32:02 STEP 2372 H=2372:00 +CPU= 10.324
  18:32:18 STEP 2373 H=2373:00 +CPU= 16.506
  18:32:28 STEP 2374 H=2374:00 +CPU= 10.192
  18:32:39 STEP 2375 H=2375:00 +CPU= 10.365
  18:32:59 STEP 2376 H=2376:00 +CPU= 19.046
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_98.zip
Uploading the intermediate file: upload_file_98.zip
  18:33:09 STEP 2377 H=2377:00 +CPU=  9.990
  18:33:19 STEP 2378 H=2378:00 +CPU= 10.372
  18:33:36 STEP 2379 H=2379:00 +CPU= 16.491
  18:33:46 STEP 2380 H=2380:00 +CPU= 10.306
  18:33:56 STEP 2381 H=2381:00 +CPU= 10.242
  18:34:13 STEP 2382 H=2382:00 +CPU= 16.696
  18:34:23 STEP 2383 H=2383:00 +CPU= 10.139
  18:34:34 STEP 2384 H=2384:00 +CPU= 10.811
  18:34:50 STEP 2385 H=2385:00 +CPU= 16.061
  18:35:00 STEP 2386 H=2386:00 +CPU= 10.400
  18:35:11 STEP 2387 H=2387:00 +CPU= 10.247
  18:35:29 STEP 2388 H=2388:00 +CPU= 17.906
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002388
  18:35:39 STEP 2389 H=2389:00 +CPU= 10.329
  18:35:49 STEP 2390 H=2390:00 +CPU= 10.420
  18:36:06 STEP 2391 H=2391:00 +CPU= 16.373
  18:36:16 STEP 2392 H=2392:00 +CPU= 10.268
  18:36:26 STEP 2393 H=2393:00 +CPU= 10.235
  18:36:43 STEP 2394 H=2394:00 +CPU= 16.536
  18:36:53 STEP 2395 H=2395:00 +CPU= 10.026
  18:37:03 STEP 2396 H=2396:00 +CPU= 10.368
  18:37:19 STEP 2397 H=2397:00 +CPU= 15.931
  18:37:29 STEP 2398 H=2398:00 +CPU= 10.310
  18:37:40 STEP 2399 H=2399:00 +CPU= 10.539
  18:38:01 STEP 2400 H=2400:00 +CPU= 18.857
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_99.zip
Uploading the intermediate file: upload_file_99.zip
  18:38:12 STEP 2401 H=2401:00 +CPU= 10.480
  18:38:22 STEP 2402 H=2402:00 +CPU= 10.263
Uploading trickle at timestep: 8640000
  18:38:38 STEP 2403 H=2403:00 +CPU= 16.417
  18:38:48 STEP 2404 H=2404:00 +CPU= 10.118
  18:38:59 STEP 2405 H=2405:00 +CPU= 10.095
  18:39:15 STEP 2406 H=2406:00 +CPU= 16.687
  18:39:25 STEP 2407 H=2407:00 +CPU=  9.957
  18:39:36 STEP 2408 H=2408:00 +CPU= 10.586
  18:39:52 STEP 2409 H=2409:00 +CPU= 16.486
  18:40:02 STEP 2410 H=2410:00 +CPU= 10.222
  18:40:13 STEP 2411 H=2411:00 +CPU= 10.685
  18:40:31 STEP 2412 H=2412:00 +CPU= 17.870
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002412
  18:40:42 STEP 2413 H=2413:00 +CPU= 10.757
  18:40:52 STEP 2414 H=2414:00 +CPU= 10.409
  18:41:09 STEP 2415 H=2415:00 +CPU= 16.640
  18:41:19 STEP 2416 H=2416:00 +CPU= 10.216
  18:41:29 STEP 2417 H=2417:00 +CPU= 10.051
  18:41:46 STEP 2418 H=2418:00 +CPU= 16.694
  18:41:56 STEP 2419 H=2419:00 +CPU= 10.042
  18:42:06 STEP 2420 H=2420:00 +CPU= 10.332
  18:42:23 STEP 2421 H=2421:00 +CPU= 16.574
  18:42:33 STEP 2422 H=2422:00 +CPU= 10.203
  18:42:43 STEP 2423 H=2423:00 +CPU= 10.368
  18:43:03 STEP 2424 H=2424:00 +CPU= 18.920
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_100.zip
Uploading the intermediate file: upload_file_100.zip
  18:43:13 STEP 2425 H=2425:00 +CPU= 10.273
  18:43:24 STEP 2426 H=2426:00 +CPU= 10.701
  18:43:40 STEP 2427 H=2427:00 +CPU= 16.315
  18:43:51 STEP 2428 H=2428:00 +CPU= 10.479
  18:44:01 STEP 2429 H=2429:00 +CPU= 10.382
  18:44:18 STEP 2430 H=2430:00 +CPU= 16.967
  18:44:28 STEP 2431 H=2431:00 +CPU= 10.155
  18:44:39 STEP 2432 H=2432:00 +CPU= 10.772
  18:44:56 STEP 2433 H=2433:00 +CPU= 16.300
  18:45:06 STEP 2434 H=2434:00 +CPU= 10.211
  18:45:16 STEP 2435 H=2435:00 +CPU= 10.273
  18:45:34 STEP 2436 H=2436:00 +CPU= 18.046
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002436
  18:45:44 STEP 2437 H=2437:00 +CPU= 10.413
  18:45:55 STEP 2438 H=2438:00 +CPU= 10.444
  18:46:11 STEP 2439 H=2439:00 +CPU= 16.258
  18:46:21 STEP 2440 H=2440:00 +CPU= 10.335
  18:46:32 STEP 2441 H=2441:00 +CPU= 10.164
  18:46:48 STEP 2442 H=2442:00 +CPU= 16.816
  18:46:58 STEP 2443 H=2443:00 +CPU=  9.946
  18:47:09 STEP 2444 H=2444:00 +CPU= 10.718
  18:47:25 STEP 2445 H=2445:00 +CPU= 16.327
  18:47:36 STEP 2446 H=2446:00 +CPU= 10.077
  18:47:46 STEP 2447 H=2447:00 +CPU= 10.687
  18:48:06 STEP 2448 H=2448:00 +CPU= 18.887
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_101.zip
Uploading the intermediate file: upload_file_101.zip
  18:48:16 STEP 2449 H=2449:00 +CPU=  9.908
  18:48:26 STEP 2450 H=2450:00 +CPU= 10.524
  18:48:42 STEP 2451 H=2451:00 +CPU= 16.213
  18:48:53 STEP 2452 H=2452:00 +CPU= 10.549
  18:49:03 STEP 2453 H=2453:00 +CPU= 10.438
  18:49:20 STEP 2454 H=2454:00 +CPU= 16.904
  18:49:30 STEP 2455 H=2455:00 +CPU= 10.195
  18:49:41 STEP 2456 H=2456:00 +CPU= 10.802
  18:49:58 STEP 2457 H=2457:00 +CPU= 16.267
  18:50:08 STEP 2458 H=2458:00 +CPU= 10.349
  18:50:18 STEP 2459 H=2459:00 +CPU= 10.327
  18:50:36 STEP 2460 H=2460:00 +CPU= 18.099
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002460
  18:50:47 STEP 2461 H=2461:00 +CPU= 10.174
  18:50:57 STEP 2462 H=2462:00 +CPU= 10.670
  18:51:14 STEP 2463 H=2463:00 +CPU= 16.424
  18:51:24 STEP 2464 H=2464:00 +CPU= 10.497
  18:51:34 STEP 2465 H=2465:00 +CPU= 10.369
  18:51:51 STEP 2466 H=2466:00 +CPU= 16.355
  18:52:01 STEP 2467 H=2467:00 +CPU= 10.417
  18:52:12 STEP 2468 H=2468:00 +CPU= 10.494
  18:52:28 STEP 2469 H=2469:00 +CPU= 16.451
  18:52:39 STEP 2470 H=2470:00 +CPU= 10.358
  18:52:49 STEP 2471 H=2471:00 +CPU= 10.543
  18:53:09 STEP 2472 H=2472:00 +CPU= 19.394
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_102.zip
Uploading the intermediate file: upload_file_102.zip
  18:53:18 STEP 2473 H=2473:00 +CPU=  9.924
  18:53:29 STEP 2474 H=2474:00 +CPU= 10.587
  18:53:45 STEP 2475 H=2475:00 +CPU= 16.144
  18:53:55 STEP 2476 H=2476:00 +CPU= 10.233
  18:54:06 STEP 2477 H=2477:00 +CPU= 10.451
  18:54:22 STEP 2478 H=2478:00 +CPU= 16.537
  18:54:33 STEP 2479 H=2479:00 +CPU= 10.660
  18:54:44 STEP 2480 H=2480:00 +CPU= 10.680
  18:55:00 STEP 2481 H=2481:00 +CPU= 16.432
  18:55:11 STEP 2482 H=2482:00 +CPU= 10.540
  18:55:21 STEP 2483 H=2483:00 +CPU=  9.975
  18:55:39 STEP 2484 H=2484:00 +CPU= 18.047
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002484
  18:55:49 STEP 2485 H=2485:00 +CPU= 10.337
  18:56:00 STEP 2486 H=2486:00 +CPU= 10.550
  18:56:16 STEP 2487 H=2487:00 +CPU= 16.638
  18:56:27 STEP 2488 H=2488:00 +CPU= 10.287
  18:56:37 STEP 2489 H=2489:00 +CPU= 10.540
  18:56:54 STEP 2490 H=2490:00 +CPU= 16.819
  18:57:04 STEP 2491 H=2491:00 +CPU= 10.566
  18:57:15 STEP 2492 H=2492:00 +CPU= 10.625
  18:57:31 STEP 2493 H=2493:00 +CPU= 16.329
  18:57:42 STEP 2494 H=2494:00 +CPU= 10.311
  18:57:52 STEP 2495 H=2495:00 +CPU= 10.405
  18:58:12 STEP 2496 H=2496:00 +CPU= 19.076
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_103.zip
  18:58:22 STEP 2497 H=2497:00 +CPU= 10.077
Uploading the intermediate file: upload_file_103.zip
  18:58:33 STEP 2498 H=2498:00 +CPU= 10.362
  18:58:49 STEP 2499 H=2499:00 +CPU= 16.149
  18:58:59 STEP 2500 H=2500:00 +CPU= 10.340
  18:59:09 STEP 2501 H=2501:00 +CPU= 10.320
  18:59:26 STEP 2502 H=2502:00 +CPU= 16.570
  18:59:36 STEP 2503 H=2503:00 +CPU= 10.038
  18:59:47 STEP 2504 H=2504:00 +CPU= 10.813
  19:00:03 STEP 2505 H=2505:00 +CPU= 16.359
  19:00:14 STEP 2506 H=2506:00 +CPU= 10.729
  19:00:24 STEP 2507 H=2507:00 +CPU= 10.288
  19:00:42 STEP 2508 H=2508:00 +CPU= 17.983
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002508
  19:00:52 STEP 2509 H=2509:00 +CPU= 10.136
  19:01:03 STEP 2510 H=2510:00 +CPU= 10.625
  19:01:19 STEP 2511 H=2511:00 +CPU= 16.514
  19:01:30 STEP 2512 H=2512:00 +CPU= 10.482
  19:01:40 STEP 2513 H=2513:00 +CPU= 10.294
  19:01:57 STEP 2514 H=2514:00 +CPU= 16.621
  19:02:07 STEP 2515 H=2515:00 +CPU= 10.067
  19:02:17 STEP 2516 H=2516:00 +CPU= 10.513
  19:02:34 STEP 2517 H=2517:00 +CPU= 16.163
  19:02:44 STEP 2518 H=2518:00 +CPU= 10.580
  19:02:55 STEP 2519 H=2519:00 +CPU= 10.420
  19:03:14 STEP 2520 H=2520:00 +CPU= 19.057
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_104.zip
Uploading the intermediate file: upload_file_104.zip
  19:03:25 STEP 2521 H=2521:00 +CPU= 10.229
  19:03:35 STEP 2522 H=2522:00 +CPU= 10.531
  19:03:51 STEP 2523 H=2523:00 +CPU= 16.182
  19:04:02 STEP 2524 H=2524:00 +CPU= 10.601
  19:04:12 STEP 2525 H=2525:00 +CPU= 10.248
  19:04:29 STEP 2526 H=2526:00 +CPU= 16.786
  19:04:39 STEP 2527 H=2527:00 +CPU= 10.091
  19:04:50 STEP 2528 H=2528:00 +CPU= 10.790
  19:05:06 STEP 2529 H=2529:00 +CPU= 16.232
  19:05:17 STEP 2530 H=2530:00 +CPU= 10.595
  19:05:27 STEP 2531 H=2531:00 +CPU= 10.175
  19:05:44 STEP 2532 H=2532:00 +CPU= 17.724
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002532
  19:05:55 STEP 2533 H=2533:00 +CPU= 10.169
  19:06:05 STEP 2534 H=2534:00 +CPU= 10.376
  19:06:21 STEP 2535 H=2535:00 +CPU= 16.247
  19:06:32 STEP 2536 H=2536:00 +CPU= 10.326
  19:06:42 STEP 2537 H=2537:00 +CPU= 10.085
  19:06:59 STEP 2538 H=2538:00 +CPU= 16.975
  19:07:09 STEP 2539 H=2539:00 +CPU=  9.866
  19:07:19 STEP 2540 H=2540:00 +CPU= 10.433
  19:07:35 STEP 2541 H=2541:00 +CPU= 16.446
  19:07:46 STEP 2542 H=2542:00 +CPU= 10.218
  19:07:56 STEP 2543 H=2543:00 +CPU= 10.687
  19:08:17 STEP 2544 H=2544:00 +CPU= 18.985
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_105.zip
Uploading the intermediate file: upload_file_105.zip
  19:08:27 STEP 2545 H=2545:00 +CPU= 10.244
  19:08:37 STEP 2546 H=2546:00 +CPU= 10.514
  19:08:54 STEP 2547 H=2547:00 +CPU= 16.303
  19:09:04 STEP 2548 H=2548:00 +CPU= 10.537
  19:09:15 STEP 2549 H=2549:00 +CPU= 10.335
  19:09:31 STEP 2550 H=2550:00 +CPU= 16.699
  19:09:41 STEP 2551 H=2551:00 +CPU= 10.024
  19:09:52 STEP 2552 H=2552:00 +CPU= 10.485
  19:10:08 STEP 2553 H=2553:00 +CPU= 16.200
  19:10:19 STEP 2554 H=2554:00 +CPU= 10.651
  19:10:29 STEP 2555 H=2555:00 +CPU= 10.352
  19:10:47 STEP 2556 H=2556:00 +CPU= 18.241
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002556
  19:10:57 STEP 2557 H=2557:00 +CPU= 10.121
  19:11:08 STEP 2558 H=2558:00 +CPU= 10.418
  19:11:24 STEP 2559 H=2559:00 +CPU= 16.316
  19:11:35 STEP 2560 H=2560:00 +CPU= 10.486
  19:11:45 STEP 2561 H=2561:00 +CPU= 10.312
  19:12:02 STEP 2562 H=2562:00 +CPU= 16.707
  19:12:12 STEP 2563 H=2563:00 +CPU= 10.106
  19:12:22 STEP 2564 H=2564:00 +CPU= 10.584
  19:12:38 STEP 2565 H=2565:00 +CPU= 16.088
  19:12:49 STEP 2566 H=2566:00 +CPU= 10.357
  19:12:59 STEP 2567 H=2567:00 +CPU= 10.463
  19:13:21 STEP 2568 H=2568:00 +CPU= 19.668
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12167149/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0505_1981050100_123_950_12167149_1_r1484203278_106.zip
Uploading the intermediate file: upload_file_106.zip
  19:13:31 STEP 2569 H=2569:00 +CPU=  9.910
  19:13:41 STEP 2570 H=2570:00 +CPU= 10.607
  19:13:58 STEP 2571 H=2571:00 +CPU= 16.322
  19:14:08 STEP 2572 H=2572:00 +CPU= 10.297
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:30:25 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 8,640,000 42,069 0.0049
09 May 2023 16:30:19 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 7,776,000 39,050 0.0050
09 May 2023 16:30:13 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 6,912,000 36,031 0.0052
09 May 2023 16:30:06 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 6,048,000 33,000 0.0055
09 May 2023 16:30:01 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 5,184,000 29,781 0.0057
09 May 2023 16:29:59 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 4,320,000 26,045 0.0060
09 May 2023 16:29:47 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 3,456,000 21,248 0.0061
09 May 2023 16:29:41 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 2,592,000 15,821 0.0061
09 May 2023 16:29:30 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 1,728,000 10,524 0.0061
09 May 2023 16:29:24 1537625 22299999 oifs_43r3_ps_0505_1981050100_123_950_12167149_1 864,000 5,142 0.0060


©2024 climateprediction.net