climateprediction.net (CPDN) home page
Task 22298294

Task 22298294

Name oifs_43r3_ps_0567_2017050100_123_986_12203211_1
Workunit 12203211
Created 15 Jan 2023, 20:23:34 UTC
Sent 15 Jan 2023, 20:30:48 UTC
Report deadline 14 Feb 2023, 20:30:48 UTC
Received 19 Jan 2023, 1:34:37 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1467890
Run time 21 hours 41 min 27 sec
CPU time 20 hours 52 min 17 sec
Validate state Invalid
Credit 8,588.16
Device peak FLOPS 4.08 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4.31 GB
Peak swap size 4.86 GB
Peak disk usage 7.50 GB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ignal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:411718:411718] [20230118:223521:1674077721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:411718:411718] [20230118:223521:1674077721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:411718:411718] [20230118:223521:1674077721:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:36:29 STEP 1896 H=1896:00 +CPU= 55.427
  22:36:54 STEP 1897 H=1897:00 +CPU= 24.754
  22:37:25 STEP 1898 H=1898:00 +CPU= 31.027
  22:38:08 STEP 1899 H=1899:00 +CPU= 42.631
  22:38:40 STEP 1900 H=1900:00 +CPU= 31.673
  22:39:07 STEP 1901 H=1901:00 +CPU= 27.761
  22:39:49 STEP 1902 H=1902:00 +CPU= 41.442
  22:40:17 STEP 1903 H=1903:00 +CPU= 28.013
  22:40:45 STEP 1904 H=1904:00 +CPU= 27.560
  22:41:30 STEP 1905 H=1905:00 +CPU= 44.998
  22:41:55 STEP 1906 H=1906:00 +CPU= 25.024
  22:42:24 STEP 1907 H=1907:00 +CPU= 29.056
  22:43:18 STEP 1908 H=1908:00 +CPU= 52.882
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001908
  22:43:49 STEP 1909 H=1909:00 +CPU= 31.007
  22:44:15 STEP 1910 H=1910:00 +CPU= 26.504
  22:44:55 STEP 1911 H=1911:00 +CPU= 39.733
  22:45:29 STEP 1912 H=1912:00 +CPU= 32.006
  22:45:54 STEP 1913 H=1913:00 +CPU= 24.966
  22:46:40 STEP 1914 H=1914:00 +CPU= 46.494
  22:47:11 STEP 1915 H=1915:00 +CPU= 30.006
  22:47:37 STEP 1916 H=1916:00 +CPU= 25.900
  22:48:21 STEP 1917 H=1917:00 +CPU= 44.681
  22:48:48 STEP 1918 H=1918:00 +CPU= 26.365
  22:49:20 STEP 1919 H=1919:00 +CPU= 31.806
  22:50:10 STEP 1920 H=1920:00 +CPU= 49.640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_79.zip
Uploading the intermediate file: upload_file_79.zip
  22:50:40 STEP 1921 H=1921:00 +CPU= 28.731
  22:51:11 STEP 1922 H=1922:00 +CPU= 31.059
  22:52:00 STEP 1923 H=1923:00 +CPU= 48.545
  22:52:26 STEP 1924 H=1924:00 +CPU= 26.335
  22:52:51 STEP 1925 H=1925:00 +CPU= 24.997
  22:53:37 STEP 1926 H=1926:00 +CPU= 45.212
  22:54:07 STEP 1927 H=1927:00 +CPU= 29.942
  22:54:38 STEP 1928 H=1928:00 +CPU= 31.652
  22:55:22 STEP 1929 H=1929:00 +CPU= 43.247
  22:55:50 STEP 1930 H=1930:00 +CPU= 28.054
  22:56:18 STEP 1931 H=1931:00 +CPU= 27.841
  22:57:02 STEP 1932 H=1932:00 +CPU= 43.630
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001932
  22:57:33 STEP 1933 H=1933:00 +CPU= 30.612
  22:57:58 STEP 1934 H=1934:00 +CPU= 25.059
  22:58:44 STEP 1935 H=1935:00 +CPU= 45.900
  22:59:15 STEP 1936 H=1936:00 +CPU= 30.978
  22:59:45 STEP 1937 H=1937:00 +CPU= 30.154
  23:00:29 STEP 1938 H=1938:00 +CPU= 43.286
  23:00:54 STEP 1939 H=1939:00 +CPU= 25.252
  23:01:25 STEP 1940 H=1940:00 +CPU= 31.010
  23:02:12 STEP 1941 H=1941:00 +CPU= 46.314
  23:02:41 STEP 1942 H=1942:00 +CPU= 28.904
  23:03:06 STEP 1943 H=1943:00 +CPU= 24.789
  23:03:58 STEP 1944 H=1944:00 +CPU= 52.017
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_80.zip
Uploading the intermediate file: upload_file_80.zip
  23:04:29 STEP 1945 H=1945:00 +CPU= 30.251
  23:04:55 STEP 1946 H=1946:00 +CPU= 25.917
  23:05:43 STEP 1947 H=1947:00 +CPU= 48.409
  23:06:10 STEP 1948 H=1948:00 +CPU= 26.778
  23:06:40 STEP 1949 H=1949:00 +CPU= 29.643
  23:07:25 STEP 1950 H=1950:00 +CPU= 44.738
  23:07:56 STEP 1951 H=1951:00 +CPU= 30.911
  23:08:27 STEP 1952 H=1952:00 +CPU= 31.187
  23:09:11 STEP 1953 H=1953:00 +CPU= 43.662
  23:09:43 STEP 1954 H=1954:00 +CPU= 31.734
  23:10:10 STEP 1955 H=1955:00 +CPU= 26.928
  23:11:01 STEP 1956 H=1956:00 +CPU= 50.667
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001956
  23:11:31 STEP 1957 H=1957:00 +CPU= 29.987
  23:12:03 STEP 1958 H=1958:00 +CPU= 31.636
  23:12:41 STEP 1959 H=1959:00 +CPU= 38.245
  23:13:07 STEP 1960 H=1960:00 +CPU= 25.342
  23:13:38 STEP 1961 H=1961:00 +CPU= 30.662
  23:14:20 STEP 1962 H=1962:00 +CPU= 42.621
  23:14:51 STEP 1963 H=1963:00 +CPU= 30.713
  23:15:22 STEP 1964 H=1964:00 +CPU= 30.817
  23:16:05 STEP 1965 H=1965:00 +CPU= 42.538
  23:16:36 STEP 1966 H=1966:00 +CPU= 31.091
  23:17:04 STEP 1967 H=1967:00 +CPU= 28.084
  23:18:00 STEP 1968 H=1968:00 +CPU= 55.052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_81.zip
Uploading the intermediate file: upload_file_81.zip
  23:18:27 STEP 1969 H=1969:00 +CPU= 26.579
  23:18:58 STEP 1970 H=1970:00 +CPU= 31.625
  23:19:42 STEP 1971 H=1971:00 +CPU= 43.262
  23:20:14 STEP 1972 H=1972:00 +CPU= 31.949
  23:20:44 STEP 1973 H=1973:00 +CPU= 30.089
  23:21:25 STEP 1974 H=1974:00 +CPU= 40.397
  23:21:56 STEP 1975 H=1975:00 +CPU= 30.827
  23:22:26 STEP 1976 H=1976:00 +CPU= 30.016
  23:23:10 STEP 1977 H=1977:00 +CPU= 43.889
  23:23:41 STEP 1978 H=1978:00 +CPU= 30.957
  23:24:08 STEP 1979 H=1979:00 +CPU= 26.248
  23:25:01 STEP 1980 H=1980:00 +CPU= 53.622
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001980
  23:25:28 STEP 1981 H=1981:00 +CPU= 26.228
  23:26:00 STEP 1982 H=1982:00 +CPU= 31.619
  23:26:39 STEP 1983 H=1983:00 +CPU= 38.817
  23:27:08 STEP 1984 H=1984:00 +CPU= 29.722
  23:27:34 STEP 1985 H=1985:00 +CPU= 25.394
  23:28:18 STEP 1986 H=1986:00 +CPU= 43.613
  23:28:44 STEP 1987 H=1987:00 +CPU= 25.832
  23:29:10 STEP 1988 H=1988:00 +CPU= 25.233
  23:29:54 STEP 1989 H=1989:00 +CPU= 43.901
  23:30:25 STEP 1990 H=1990:00 +CPU= 30.631
  23:30:54 STEP 1991 H=1991:00 +CPU= 29.015
  23:31:45 STEP 1992 H=1992:00 +CPU= 46.101
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_82.zip
Uploading the intermediate file: upload_file_82.zip
  23:32:10 STEP 1993 H=1993:00 +CPU= 25.047
  23:32:42 STEP 1994 H=1994:00 +CPU= 31.224
  23:33:29 STEP 1995 H=1995:00 +CPU= 47.496
  23:34:02 STEP 1996 H=1996:00 +CPU= 32.179
  23:34:32 STEP 1997 H=1997:00 +CPU= 29.853
  23:35:13 STEP 1998 H=1998:00 +CPU= 40.986
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0567
(argv4) batchid: 986
(argv5) wuid: 12203211
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0c10adc1efc9b437a0c7971bd1855831 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0567_2017050100_123_986_12203211.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0567_2017050100_123_986_12203211.zip
ic_ancil_file: ic_ancil_12203211
ifsdata_file: ifsdata_12203211
climate_data_file: clim_data_12203211
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cf6834cb101636b6c9760a480d887eea to: /var/lib/boinc-client/slots/2/ic_ancil_12203211.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203211.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0dc113fcd96cca879d22465953636711 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203211.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203211.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203211.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203211.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203211.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203211.xml
-- Model is restarting --
Adjusting last_iter, 1999, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203211.xml
last_cpu_time: 66824
upload_file_number: 83
last_iter: 1991
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850
The child process has been launched with process id: 414862
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:414862:414862] [20230118:233643:1674081403:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:37:48 STEP 1992 H=1992:00 +CPU= 54.429
  23:38:18 STEP 1993 H=1993:00 +CPU= 30.077
  23:38:49 STEP 1994 H=1994:00 +CPU= 30.759
  23:39:28 STEP 1995 H=1995:00 +CPU= 38.230
  23:39:55 STEP 1996 H=1996:00 +CPU= 27.487
  23:40:26 STEP 1997 H=1997:00 +CPU= 30.516
  23:41:12 STEP 1998 H=1998:00 +CPU= 46.060
  23:41:43 STEP 1999 H=1999:00 +CPU= 30.751
  23:42:14 STEP 2000 H=2000:00 +CPU= 30.806
  23:43:03 STEP 2001 H=2001:00 +CPU= 48.660
  23:43:28 STEP 2002 H=2002:00 +CPU= 25.425
  23:43:54 STEP 2003 H=2003:00 +CPU= 25.197
  23:44:48 STEP 2004 H=2004:00 +CPU= 53.590
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002004
  23:45:12 STEP 2005 H=2005:00 +CPU= 24.689
  23:45:44 STEP 2006 H=2006:00 +CPU= 31.609
  23:46:32 STEP 2007 H=2007:00 +CPU= 48.003
  23:47:03 STEP 2008 H=2008:00 +CPU= 30.452
  23:47:27 STEP 2009 H=2009:00 +CPU= 24.426
  23:48:12 STEP 2010 H=2010:00 +CPU= 44.325
  23:48:42 STEP 2011 H=2011:00 +CPU= 29.887
  23:49:09 STEP 2012 H=2012:00 +CPU= 26.798
  23:49:57 STEP 2013 H=2013:00 +CPU= 47.070
  23:50:23 STEP 2014 H=2014:00 +CPU= 25.836
  23:50:51 STEP 2015 H=2015:00 +CPU= 28.330
  23:51:46 STEP 2016 H=2016:00 +CPU= 53.004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_83.zip
Uploading the intermediate file: upload_file_83.zip
  23:52:15 STEP 2017 H=2017:00 +CPU= 28.874
  23:52:46 STEP 2018 H=2018:00 +CPU= 30.895
  23:53:28 STEP 2019 H=2019:00 +CPU= 41.180
  23:54:00 STEP 2020 H=2020:00 +CPU= 31.741
  23:54:25 STEP 2021 H=2021:00 +CPU= 25.040
  23:55:12 STEP 2022 H=2022:00 +CPU= 46.245
  23:55:39 STEP 2023 H=2023:00 +CPU= 27.563
  23:56:05 STEP 2024 H=2024:00 +CPU= 25.490
  23:56:53 STEP 2025 H=2025:00 +CPU= 47.696
  23:57:19 STEP 2026 H=2026:00 +CPU= 25.584
  23:57:47 STEP 2027 H=2027:00 +CPU= 28.288
  23:58:40 STEP 2028 H=2028:00 +CPU= 52.660
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002028
  23:59:11 STEP 2029 H=2029:00 +CPU= 30.842
  23:59:42 STEP 2030 H=2030:00 +CPU= 30.891
  00:00:27 STEP 2031 H=2031:00 +CPU= 44.408
  00:00:58 STEP 2032 H=2032:00 +CPU= 31.476
  00:01:23 STEP 2033 H=2033:00 +CPU= 24.908
  00:02:12 STEP 2034 H=2034:00 +CPU= 48.575
  00:02:37 STEP 2035 H=2035:00 +CPU= 24.273
  00:03:06 STEP 2036 H=2036:00 +CPU= 29.617
  00:03:52 STEP 2037 H=2037:00 +CPU= 45.163
  00:04:19 STEP 2038 H=2038:00 +CPU= 26.918
  00:04:50 STEP 2039 H=2039:00 +CPU= 31.230
  00:05:40 STEP 2040 H=2040:00 +CPU= 49.812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_84.zip
Uploading the intermediate file: upload_file_84.zip
  00:06:11 STEP 2041 H=2041:00 +CPU= 30.632
  00:06:40 STEP 2042 H=2042:00 +CPU= 28.623
  00:07:25 STEP 2043 H=2043:00 +CPU= 44.226
  00:07:55 STEP 2044 H=2044:00 +CPU= 30.028
  00:08:23 STEP 2045 H=2045:00 +CPU= 27.901
  00:09:06 STEP 2046 H=2046:00 +CPU= 42.423
  00:09:31 STEP 2047 H=2047:00 +CPU= 25.814
  00:10:03 STEP 2048 H=2048:00 +CPU= 31.826
  00:10:45 STEP 2049 H=2049:00 +CPU= 41.815
  00:11:16 STEP 2050 H=2050:00 +CPU= 29.895
  00:11:46 STEP 2051 H=2051:00 +CPU= 29.910
  00:12:37 STEP 2052 H=2052:00 +CPU= 50.216
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002052
  00:13:07 STEP 2053 H=2053:00 +CPU= 30.027
  00:13:33 STEP 2054 H=2054:00 +CPU= 25.727
  00:14:21 STEP 2055 H=2055:00 +CPU= 48.383
  00:14:52 STEP 2056 H=2056:00 +CPU= 30.937
  00:15:23 STEP 2057 H=2057:00 +CPU= 30.709
  00:16:02 STEP 2058 H=2058:00 +CPU= 38.679
  00:16:30 STEP 2059 H=2059:00 +CPU= 27.918
  00:17:01 STEP 2060 H=2060:00 +CPU= 30.907
  00:17:42 STEP 2061 H=2061:00 +CPU= 40.378
  00:18:14 STEP 2062 H=2062:00 +CPU= 31.797
  00:18:43 STEP 2063 H=2063:00 +CPU= 29.593
  00:19:37 STEP 2064 H=2064:00 +CPU= 53.466
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_85.zip
Uploading the intermediate file: upload_file_85.zip
  00:20:07 STEP 2065 H=2065:00 +CPU= 30.087
  00:20:39 STEP 2066 H=2066:00 +CPU= 31.558
  00:21:25 STEP 2067 H=2067:00 +CPU= 45.231
  00:21:50 STEP 2068 H=2068:00 +CPU= 25.527
  00:22:21 STEP 2069 H=2069:00 +CPU= 30.969
  00:23:04 STEP 2070 H=2070:00 +CPU= 42.368
  00:23:35 STEP 2071 H=2071:00 +CPU= 30.490
  00:24:06 STEP 2072 H=2072:00 +CPU= 30.924
  00:24:50 STEP 2073 H=2073:00 +CPU= 43.818
  00:25:21 STEP 2074 H=2074:00 +CPU= 30.928
  00:25:51 STEP 2075 H=2075:00 +CPU= 30.048
  00:26:42 STEP 2076 H=2076:00 +CPU= 50.353
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002076
  00:27:06 STEP 2077 H=2077:00 +CPU= 24.646
  00:27:38 STEP 2078 H=2078:00 +CPU= 31.333
  00:28:26 STEP 2079 H=2079:00 +CPU= 47.780
  00:28:58 STEP 2080 H=2080:00 +CPU= 31.736
  00:29:25 STEP 2081 H=2081:00 +CPU= 27.210
  00:30:08 STEP 2082 H=2082:00 +CPU= 42.553
  00:30:35 STEP 2083 H=2083:00 +CPU= 27.058
  00:31:01 STEP 2084 H=2084:00 +CPU= 25.754
  00:31:46 STEP 2085 H=2085:00 +CPU= 45.097
  00:32:17 STEP 2086 H=2086:00 +CPU= 31.023
  00:32:49 STEP 2087 H=2087:00 +CPU= 31.452
  00:33:49 STEP 2088 H=2088:00 +CPU= 55.880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_86.zip
Uploading the intermediate file: upload_file_86.zip
  00:34:13 STEP 2089 H=2089:00 +CPU= 24.591
  00:34:45 STEP 2090 H=2090:00 +CPU= 31.852
  00:35:28 STEP 2091 H=2091:00 +CPU= 42.705
  00:36:00 STEP 2092 H=2092:00 +CPU= 31.622
  00:36:25 STEP 2093 H=2093:00 +CPU= 25.064
  00:37:08 STEP 2094 H=2094:00 +CPU= 42.293
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0567
(argv4) batchid: 986
(argv5) wuid: 12203211
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0c10adc1efc9b437a0c7971bd1855831 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0567_2017050100_123_986_12203211.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0567_2017050100_123_986_12203211.zip
ic_ancil_file: ic_ancil_12203211
ifsdata_file: ifsdata_12203211
climate_data_file: clim_data_12203211
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cf6834cb101636b6c9760a480d887eea to: /var/lib/boinc-client/slots/2/ic_ancil_12203211.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203211.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0dc113fcd96cca879d22465953636711 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203211.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203211.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203211.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203211.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203211.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203211.xml
-- Model is restarting --
Adjusting last_iter, 2095, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203211.xml
last_cpu_time: 70426
upload_file_number: 87
last_iter: 2087
last_upload: 7516800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850
The child process has been launched with process id: 417950
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:417950:417950] [20230119:003834:1674085114:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:39:40 STEP 2088 H=2088:00 +CPU= 53.928
  00:40:07 STEP 2089 H=2089:00 +CPU= 27.094
  00:40:38 STEP 2090 H=2090:00 +CPU= 31.109
  00:41:26 STEP 2091 H=2091:00 +CPU= 48.044
  00:41:55 STEP 2092 H=2092:00 +CPU= 28.170
  00:42:20 STEP 2093 H=2093:00 +CPU= 24.931
  00:43:09 STEP 2094 H=2094:00 +CPU= 48.712
  00:43:39 STEP 2095 H=2095:00 +CPU= 29.980
  00:44:05 STEP 2096 H=2096:00 +CPU= 26.536
  00:44:54 STEP 2097 H=2097:00 +CPU= 47.908
  00:45:25 STEP 2098 H=2098:00 +CPU= 31.166
  00:45:56 STEP 2099 H=2099:00 +CPU= 31.086
  00:46:38 STEP 2100 H=2100:00 +CPU= 42.071
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002100
  00:47:09 STEP 2101 H=2101:00 +CPU= 30.811
  00:47:40 STEP 2102 H=2102:00 +CPU= 30.928
  00:48:26 STEP 2103 H=2103:00 +CPU= 45.188
  00:48:53 STEP 2104 H=2104:00 +CPU= 27.055
  00:49:22 STEP 2105 H=2105:00 +CPU= 29.035
  00:50:06 STEP 2106 H=2106:00 +CPU= 43.698
  00:50:35 STEP 2107 H=2107:00 +CPU= 29.040
  00:51:03 STEP 2108 H=2108:00 +CPU= 28.083
  00:51:52 STEP 2109 H=2109:00 +CPU= 48.118
  00:52:23 STEP 2110 H=2110:00 +CPU= 31.353
  00:52:53 STEP 2111 H=2111:00 +CPU= 29.478
  00:53:42 STEP 2112 H=2112:00 +CPU= 48.292
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_87.zip
Uploading the intermediate file: upload_file_87.zip
  00:54:12 STEP 2113 H=2113:00 +CPU= 30.449
  00:54:39 STEP 2114 H=2114:00 +CPU= 26.434
  00:55:26 STEP 2115 H=2115:00 +CPU= 46.807
  00:55:57 STEP 2116 H=2116:00 +CPU= 31.039
  00:56:28 STEP 2117 H=2117:00 +CPU= 30.622
  00:57:11 STEP 2118 H=2118:00 +CPU= 42.735
  00:57:36 STEP 2119 H=2119:00 +CPU= 25.457
  00:58:09 STEP 2120 H=2120:00 +CPU= 31.998
  00:58:57 STEP 2121 H=2121:00 +CPU= 47.909
  00:59:29 STEP 2122 H=2122:00 +CPU= 31.868
  00:59:54 STEP 2123 H=2123:00 +CPU= 24.891
  01:00:42 STEP 2124 H=2124:00 +CPU= 47.566
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002124
  01:01:12 STEP 2125 H=2125:00 +CPU= 30.025
  01:01:43 STEP 2126 H=2126:00 +CPU= 31.322
  01:02:27 STEP 2127 H=2127:00 +CPU= 43.828
  01:02:53 STEP 2128 H=2128:00 +CPU= 25.350
  01:03:22 STEP 2129 H=2129:00 +CPU= 29.250
  01:04:11 STEP 2130 H=2130:00 +CPU= 48.733
  01:04:42 STEP 2131 H=2131:00 +CPU= 30.805
  01:05:08 STEP 2132 H=2132:00 +CPU= 26.179
  01:05:51 STEP 2133 H=2133:00 +CPU= 42.892
  01:06:23 STEP 2134 H=2134:00 +CPU= 30.892
  01:06:51 STEP 2135 H=2135:00 +CPU= 28.059
  01:07:46 STEP 2136 H=2136:00 +CPU= 53.223
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_88.zip
Uploading the intermediate file: upload_file_88.zip
  01:08:16 STEP 2137 H=2137:00 +CPU= 29.404
  01:08:48 STEP 2138 H=2138:00 +CPU= 32.045
  01:09:30 STEP 2139 H=2139:00 +CPU= 41.481
  01:10:00 STEP 2140 H=2140:00 +CPU= 30.108
  01:10:26 STEP 2141 H=2141:00 +CPU= 25.576
  01:11:09 STEP 2142 H=2142:00 +CPU= 43.213
  01:11:35 STEP 2143 H=2143:00 +CPU= 25.798
  01:12:07 STEP 2144 H=2144:00 +CPU= 31.204
  01:12:52 STEP 2145 H=2145:00 +CPU= 44.824
  01:13:17 STEP 2146 H=2146:00 +CPU= 25.592
  01:13:46 STEP 2147 H=2147:00 +CPU= 28.885
  01:14:40 STEP 2148 H=2148:00 +CPU= 52.929
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002148
  01:15:11 STEP 2149 H=2149:00 +CPU= 30.820
  01:15:37 STEP 2150 H=2150:00 +CPU= 26.142
  01:16:18 STEP 2151 H=2151:00 +CPU= 41.178
  01:16:50 STEP 2152 H=2152:00 +CPU= 31.787
  01:17:18 STEP 2153 H=2153:00 +CPU= 27.860
  01:18:05 STEP 2154 H=2154:00 +CPU= 46.885
  01:18:35 STEP 2155 H=2155:00 +CPU= 30.038
  01:19:07 STEP 2156 H=2156:00 +CPU= 31.543
  01:19:50 STEP 2157 H=2157:00 +CPU= 43.233
  01:20:19 STEP 2158 H=2158:00 +CPU= 28.305
  01:20:51 STEP 2159 H=2159:00 +CPU= 31.569
  01:21:39 STEP 2160 H=2160:00 +CPU= 47.544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_89.zip
Uploading the intermediate file: upload_file_89.zip
  01:22:10 STEP 2161 H=2161:00 +CPU= 31.142
  01:22:40 STEP 2162 H=2162:00 +CPU= 30.096
  01:23:24 STEP 2163 H=2163:00 +CPU= 43.298
  01:23:52 STEP 2164 H=2164:00 +CPU= 27.582
  01:24:17 STEP 2165 H=2165:00 +CPU= 25.302
  01:25:07 STEP 2166 H=2166:00 +CPU= 49.656
  01:25:37 STEP 2167 H=2167:00 +CPU= 30.030
  01:26:08 STEP 2168 H=2168:00 +CPU= 31.128
  01:26:47 STEP 2169 H=2169:00 +CPU= 38.326
  01:27:16 STEP 2170 H=2170:00 +CPU= 29.024
  01:27:46 STEP 2171 H=2171:00 +CPU= 30.350
  01:28:32 STEP 2172 H=2172:00 +CPU= 45.797
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002172
  01:29:04 STEP 2173 H=2173:00 +CPU= 31.325
  01:29:32 STEP 2174 H=2174:00 +CPU= 28.183
  01:30:16 STEP 2175 H=2175:00 +CPU= 44.019
  01:30:44 STEP 2176 H=2176:00 +CPU= 27.902
  01:31:12 STEP 2177 H=2177:00 +CPU= 27.521
  01:31:59 STEP 2178 H=2178:00 +CPU= 46.731
  01:32:29 STEP 2179 H=2179:00 +CPU= 29.396
  01:33:01 STEP 2180 H=2180:00 +CPU= 31.978
  01:33:44 STEP 2181 H=2181:00 +CPU= 42.644
  01:34:16 STEP 2182 H=2182:00 +CPU= 31.984
  01:34:47 STEP 2183 H=2183:00 +CPU= 31.038
  01:35:38 STEP 2184 H=2184:00 +CPU= 49.962
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_90.zip
Uploading the intermediate file: upload_file_90.zip
  01:36:08 STEP 2185 H=2185:00 +CPU= 30.158
  01:36:36 STEP 2186 H=2186:00 +CPU= 28.354
  01:37:24 STEP 2187 H=2187:00 +CPU= 46.910
  01:37:48 STEP 2188 H=2188:00 +CPU= 24.803
  01:38:15 STEP 2189 H=2189:00 +CPU= 26.446
  01:39:05 STEP 2190 H=2190:00 +CPU= 49.462
  01:39:35 STEP 2191 H=2191:00 +CPU= 30.240
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0567
(argv4) batchid: 986
(argv5) wuid: 12203211
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0c10adc1efc9b437a0c7971bd1855831 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0567_2017050100_123_986_12203211.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0567_2017050100_123_986_12203211.zip
ic_ancil_file: ic_ancil_12203211
ifsdata_file: ifsdata_12203211
climate_data_file: clim_data_12203211
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cf6834cb101636b6c9760a480d887eea to: /var/lib/boinc-client/slots/2/ic_ancil_12203211.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203211.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0dc113fcd96cca879d22465953636711 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203211.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203211.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203211.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203211.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203211.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203211.xml
-- Model is restarting --
Adjusting last_iter, 2192, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203211.xml
last_cpu_time: 74068
upload_file_number: 91
last_iter: 2183
last_upload: 7862400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850
The child process has been launched with process id: 421355
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:421355:421355] [20230119:014054:1674088854:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:41:53 STEP 2184 H=2184:00 +CPU= 45.936
  01:42:21 STEP 2185 H=2185:00 +CPU= 28.660
  01:42:53 STEP 2186 H=2186:00 +CPU= 31.715
  01:43:41 STEP 2187 H=2187:00 +CPU= 47.966
  01:44:13 STEP 2188 H=2188:00 +CPU= 31.600
  01:44:40 STEP 2189 H=2189:00 +CPU= 26.332
  01:45:24 STEP 2190 H=2190:00 +CPU= 44.383
  01:45:54 STEP 2191 H=2191:00 +CPU= 29.549
  01:46:18 STEP 2192 H=2192:00 +CPU= 24.334
  01:47:06 STEP 2193 H=2193:00 +CPU= 47.490
  01:47:33 STEP 2194 H=2194:00 +CPU= 26.205
  01:48:00 STEP 2195 H=2195:00 +CPU= 27.515
  01:48:53 STEP 2196 H=2196:00 +CPU= 52.535
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002196
  01:49:18 STEP 2197 H=2197:00 +CPU= 24.739
  01:49:50 STEP 2198 H=2198:00 +CPU= 32.422
  01:50:39 STEP 2199 H=2199:00 +CPU= 47.953
  01:51:06 STEP 2200 H=2200:00 +CPU= 27.587
  01:51:32 STEP 2201 H=2201:00 +CPU= 25.473
  01:52:13 STEP 2202 H=2202:00 +CPU= 41.141
  01:52:44 STEP 2203 H=2203:00 +CPU= 30.560
  01:53:15 STEP 2204 H=2204:00 +CPU= 30.855
  01:54:01 STEP 2205 H=2205:00 +CPU= 45.484
  01:54:26 STEP 2206 H=2206:00 +CPU= 25.569
  01:54:54 STEP 2207 H=2207:00 +CPU= 27.055
  01:55:52 STEP 2208 H=2208:00 +CPU= 55.878
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203211/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0567_2017050100_123_986_12203211_1_r40450850_91.zip
Uploading the intermediate file: upload_file_91.zip
  01:56:20 STEP 2209 H=2209:00 +CPU= 28.052
  01:56:52 STEP 2210 H=2210:00 +CPU= 32.081
  01:57:36 STEP 2211 H=2211:00 +CPU= 42.906
  01:58:08 STEP 2212 H=2212:00 +CPU= 31.875
  01:58:36 STEP 2213 H=2213:00 +CPU= 28.478
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:25:40 1467890 22298294 oifs_43r3_ps_0567_2017050100_123_986_12203211_1 6,134,400 55,881 0.0091
09 May 2023 16:25:28 1467890 22298294 oifs_43r3_ps_0567_2017050100_123_986_12203211_1 5,270,400 48,202 0.0091
09 May 2023 16:25:18 1467890 22298294 oifs_43r3_ps_0567_2017050100_123_986_12203211_1 4,406,400 40,474 0.0092
09 May 2023 16:25:06 1467890 22298294 oifs_43r3_ps_0567_2017050100_123_986_12203211_1 3,542,400 32,795 0.0093


©2024 cpdn.org