climateprediction.net home page
Task 22302803

Task 22302803

Name oifs_43r3_ps_0926_2001050100_123_970_12187570_2
Workunit 12187570
Created 24 Jan 2023, 8:07:36 UTC
Sent 24 Jan 2023, 8:21:36 UTC
Report deadline 25 Mar 2023, 8:21:36 UTC
Received 30 Jan 2023, 20:41:07 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1506016
Run time 22 hours 37 min 54 sec
CPU time 19 hours 12 min 20 sec
Validate state Invalid
Credit 9,192.96
Device peak FLOPS 5.52 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.49 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,682.18 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:144643:1674859603:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:47:55 STEP 1920 H=1920:00 +CPU= 43.958
  14:48:22 STEP 1921 H=1921:00 +CPU= 26.304
  14:48:49 STEP 1922 H=1922:00 +CPU= 27.100
  14:49:31 STEP 1923 H=1923:00 +CPU= 37.889
  14:50:00 STEP 1924 H=1924:00 +CPU= 26.571
  14:50:24 STEP 1925 H=1925:00 +CPU= 22.753
  14:51:04 STEP 1926 H=1926:00 +CPU= 38.586
  14:51:31 STEP 1927 H=1927:00 +CPU= 26.281
  14:51:58 STEP 1928 H=1928:00 +CPU= 26.205
  14:52:42 STEP 1929 H=1929:00 +CPU= 38.591
  14:53:07 STEP 1930 H=1930:00 +CPU= 24.132
  14:53:31 STEP 1931 H=1931:00 +CPU= 22.535
  14:54:13 STEP 1932 H=1932:00 +CPU= 40.335
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001932
  14:54:41 STEP 1933 H=1933:00 +CPU= 25.513
  14:55:09 STEP 1934 H=1934:00 +CPU= 26.560
  14:55:50 STEP 1935 H=1935:00 +CPU= 40.894
  14:56:13 STEP 1936 H=1936:00 +CPU= 22.703
  14:56:40 STEP 1937 H=1937:00 +CPU= 24.749
  14:57:19 STEP 1938 H=1938:00 +CPU= 37.188
  14:57:45 STEP 1939 H=1939:00 +CPU= 24.254
  14:58:12 STEP 1940 H=1940:00 +CPU= 27.086
  14:58:54 STEP 1941 H=1941:00 +CPU= 40.693
  14:59:22 STEP 1942 H=1942:00 +CPU= 26.773
  14:59:48 STEP 1943 H=1943:00 +CPU= 26.150
  15:00:35 STEP 1944 H=1944:00 +CPU= 46.170
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_80.zip
Uploading the intermediate file: upload_file_80.zip
  15:01:02 STEP 1945 H=1945:00 +CPU= 26.103
  15:01:25 STEP 1946 H=1946:00 +CPU= 21.985
  15:02:05 STEP 1947 H=1947:00 +CPU= 40.243
  15:02:27 STEP 1948 H=1948:00 +CPU= 21.629
  15:02:48 STEP 1949 H=1949:00 +CPU= 20.962
  15:03:22 STEP 1950 H=1950:00 +CPU= 33.364
  15:03:43 STEP 1951 H=1951:00 +CPU= 20.978
  15:04:09 STEP 1952 H=1952:00 +CPU= 24.373
  15:04:45 STEP 1953 H=1953:00 +CPU= 35.404
  15:05:07 STEP 1954 H=1954:00 +CPU= 21.651
  15:05:28 STEP 1955 H=1955:00 +CPU= 20.979
  15:06:05 STEP 1956 H=1956:00 +CPU= 36.322
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001956
  15:06:26 STEP 1957 H=1957:00 +CPU= 20.987
  15:06:48 STEP 1958 H=1958:00 +CPU= 21.668
  15:07:21 STEP 1959 H=1959:00 +CPU= 32.587
  15:07:43 STEP 1960 H=1960:00 +CPU= 21.643
  15:08:04 STEP 1961 H=1961:00 +CPU= 20.968
  15:08:45 STEP 1962 H=1962:00 +CPU= 40.111
  15:09:11 STEP 1963 H=1963:00 +CPU= 26.266
  15:09:38 STEP 1964 H=1964:00 +CPU= 27.056
  15:10:20 STEP 1965 H=1965:00 +CPU= 40.654
  15:10:47 STEP 1966 H=1966:00 +CPU= 26.968
  15:11:14 STEP 1967 H=1967:00 +CPU= 27.084
  15:12:00 STEP 1968 H=1968:00 +CPU= 45.706
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_81.zip
Uploading the intermediate file: upload_file_81.zip
  15:12:25 STEP 1969 H=1969:00 +CPU= 24.429
  15:12:52 STEP 1970 H=1970:00 +CPU= 26.977
  15:13:30 STEP 1971 H=1971:00 +CPU= 37.085
  15:13:50 STEP 1972 H=1972:00 +CPU= 20.294
  15:14:09 STEP 1973 H=1973:00 +CPU= 18.579
  15:14:49 STEP 1974 H=1974:00 +CPU= 38.642
  15:15:15 STEP 1975 H=1975:00 +CPU= 26.236
  15:15:42 STEP 1976 H=1976:00 +CPU= 27.065
  15:16:23 STEP 1977 H=1977:00 +CPU= 40.730
  15:16:51 STEP 1978 H=1978:00 +CPU= 27.025
  15:17:17 STEP 1979 H=1979:00 +CPU= 26.252
  15:18:03 STEP 1980 H=1980:00 +CPU= 45.161
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001980
  15:18:29 STEP 1981 H=1981:00 +CPU= 26.236
  15:18:56 STEP 1982 H=1982:00 +CPU= 27.036
  15:19:38 STEP 1983 H=1983:00 +CPU= 40.575
  15:20:05 STEP 1984 H=1984:00 +CPU= 27.030
  15:20:32 STEP 1985 H=1985:00 +CPU= 26.229
  15:21:13 STEP 1986 H=1986:00 +CPU= 41.511
  15:21:40 STEP 1987 H=1987:00 +CPU= 26.195
  15:22:07 STEP 1988 H=1988:00 +CPU= 26.966
  15:22:48 STEP 1989 H=1989:00 +CPU= 40.598
  15:23:15 STEP 1990 H=1990:00 +CPU= 26.975
  15:23:42 STEP 1991 H=1991:00 +CPU= 27.102
  15:24:31 STEP 1992 H=1992:00 +CPU= 48.299
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_82.zip
Uploading the intermediate file: upload_file_82.zip
  15:24:58 STEP 1993 H=1993:00 +CPU= 26.062
  15:25:25 STEP 1994 H=1994:00 +CPU= 27.098
  15:26:06 STEP 1995 H=1995:00 +CPU= 40.655
  15:26:33 STEP 1996 H=1996:00 +CPU= 26.277
  15:26:57 STEP 1997 H=1997:00 +CPU= 24.026
  15:27:39 STEP 1998 H=1998:00 +CPU= 41.567
  15:28:05 STEP 1999 H=1999:00 +CPU= 26.221
  15:28:33 STEP 2000 H=2000:00 +CPU= 27.032
  15:29:14 STEP 2001 H=2001:00 +CPU= 40.587
  15:29:41 STEP 2002 H=2002:00 +CPU= 26.990
  15:30:07 STEP 2003 H=2003:00 +CPU= 26.206
  15:30:53 STEP 2004 H=2004:00 +CPU= 45.065
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002004
  15:31:19 STEP 2005 H=2005:00 +CPU= 26.176
  15:31:46 STEP 2006 H=2006:00 +CPU= 26.955
  15:32:27 STEP 2007 H=2007:00 +CPU= 40.557
  15:32:49 STEP 2008 H=2008:00 +CPU= 21.930
  15:33:11 STEP 2009 H=2009:00 +CPU= 21.029
  15:33:48 STEP 2010 H=2010:00 +CPU= 36.920
  15:34:14 STEP 2011 H=2011:00 +CPU= 26.199
  15:34:41 STEP 2012 H=2012:00 +CPU= 26.941
  15:35:22 STEP 2013 H=2013:00 +CPU= 40.631
  15:35:50 STEP 2014 H=2014:00 +CPU= 26.942
  15:36:17 STEP 2015 H=2015:00 +CPU= 27.049
  15:37:05 STEP 2016 H=2016:00 +CPU= 47.716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_83.zip
Uploading the intermediate file: upload_file_83.zip
  15:37:32 STEP 2017 H=2017:00 +CPU= 26.062
  15:37:54 STEP 2018 H=2018:00 +CPU= 21.891
  15:38:27 STEP 2019 H=2019:00 +CPU= 32.703
  15:38:50 STEP 2020 H=2020:00 +CPU= 22.635
  15:39:14 STEP 2021 H=2021:00 +CPU= 23.536
  15:39:56 STEP 2022 H=2022:00 +CPU= 41.476
  15:40:22 STEP 2023 H=2023:00 +CPU= 26.177
  15:40:50 STEP 2024 H=2024:00 +CPU= 26.973
  15:41:31 STEP 2025 H=2025:00 +CPU= 40.590
  15:41:54 STEP 2026 H=2026:00 +CPU= 23.279
  15:42:16 STEP 2027 H=2027:00 +CPU= 21.529
  15:43:02 STEP 2028 H=2028:00 +CPU= 45.042
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002028
  15:43:29 STEP 2029 H=2029:00 +CPU= 26.199
  15:43:55 STEP 2030 H=2030:00 +CPU= 25.256
  15:44:32 STEP 2031 H=2031:00 +CPU= 35.663
  15:44:57 STEP 2032 H=2032:00 +CPU= 25.134
  15:45:23 STEP 2033 H=2033:00 +CPU= 25.978
  15:46:05 STEP 2034 H=2034:00 +CPU= 41.501
  15:46:31 STEP 2035 H=2035:00 +CPU= 25.287
  15:46:59 STEP 2036 H=2036:00 +CPU= 26.862
  15:47:39 STEP 2037 H=2037:00 +CPU= 38.606
  15:48:06 STEP 2038 H=2038:00 +CPU= 24.779
  15:48:34 STEP 2039 H=2039:00 +CPU= 26.937
  15:49:24 STEP 2040 H=2040:00 +CPU= 48.267
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_84.zip
Uploading the intermediate file: upload_file_84.zip
  15:49:50 STEP 2041 H=2041:00 +CPU= 25.089
  15:50:15 STEP 2042 H=2042:00 +CPU= 23.966
  15:50:56 STEP 2043 H=2043:00 +CPU= 40.004
  15:51:21 STEP 2044 H=2044:00 +CPU= 24.186
  15:51:45 STEP 2045 H=2045:00 +CPU= 24.045
  15:52:22 STEP 2046 H=2046:00 +CPU= 35.552
  15:52:49 STEP 2047 H=2047:00 +CPU= 26.186
  15:53:17 STEP 2048 H=2048:00 +CPU= 26.821
  15:53:54 STEP 2049 H=2049:00 +CPU= 36.437
  15:54:18 STEP 2050 H=2050:00 +CPU= 23.039
  15:54:43 STEP 2051 H=2051:00 +CPU= 24.057
  15:55:21 STEP 2052 H=2052:00 +CPU= 37.728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002052
  15:55:48 STEP 2053 H=2053:00 +CPU= 25.686
  15:56:16 STEP 2054 H=2054:00 +CPU= 26.753
  15:56:59 STEP 2055 H=2055:00 +CPU= 39.562
  15:57:26 STEP 2056 H=2056:00 +CPU= 25.416
  15:57:53 STEP 2057 H=2057:00 +CPU= 24.729
  15:58:39 STEP 2058 H=2058:00 +CPU= 40.800
  15:59:05 STEP 2059 H=2059:00 +CPU= 25.133
  15:59:33 STEP 2060 H=2060:00 +CPU= 26.638
  16:00:14 STEP 2061 H=2061:00 +CPU= 40.325
  16:00:41 STEP 2062 H=2062:00 +CPU= 25.684
  16:01:05 STEP 2063 H=2063:00 +CPU= 23.911
  16:01:55 STEP 2064 H=2064:00 +CPU= 48.167
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_85.zip
Uploading the intermediate file: upload_file_85.zip
  16:02:19 STEP 2065 H=2065:00 +CPU= 23.021
  16:02:45 STEP 2066 H=2066:00 +CPU= 25.576
  16:03:22 STEP 2067 H=2067:00 +CPU= 36.202
  16:03:46 STEP 2068 H=2068:00 +CPU= 23.644
  16:04:11 STEP 2069 H=2069:00 +CPU= 25.124
  16:04:45 STEP 2070 H=2070:00 +CPU= 33.552
  16:05:10 STEP 2071 H=2071:00 +CPU= 24.364
  16:05:37 STEP 2072 H=2072:00 +CPU= 26.889
  16:06:18 STEP 2073 H=2073:00 +CPU= 40.410
  16:06:42 STEP 2074 H=2074:00 +CPU= 24.073
  16:07:04 STEP 2075 H=2075:00 +CPU= 21.140
  16:07:40 STEP 2076 H=2076:00 +CPU= 36.523
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002076
  16:08:02 STEP 2077 H=2077:00 +CPU= 21.145
  16:08:24 STEP 2078 H=2078:00 +CPU= 21.811
  16:08:57 STEP 2079 H=2079:00 +CPU= 32.772
  16:09:19 STEP 2080 H=2080:00 +CPU= 21.819
  16:09:40 STEP 2081 H=2081:00 +CPU= 21.141
  16:10:18 STEP 2082 H=2082:00 +CPU= 37.488
  16:10:43 STEP 2083 H=2083:00 +CPU= 23.857
  16:11:07 STEP 2084 H=2084:00 +CPU= 23.868
  16:11:49 STEP 2085 H=2085:00 +CPU= 40.322
  16:12:16 STEP 2086 H=2086:00 +CPU= 25.699
  16:12:41 STEP 2087 H=2087:00 +CPU= 24.328
  16:13:29 STEP 2088 H=2088:00 +CPU= 46.992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_86.zip
Uploading the intermediate file: upload_file_86.zip
  16:13:55 STEP 2089 H=2089:00 +CPU= 25.950
  16:14:23 STEP 2090 H=2090:00 +CPU= 26.704
  16:15:02 STEP 2091 H=2091:00 +CPU= 39.169
  16:15:30 STEP 2092 H=2092:00 +CPU= 26.842
  16:15:54 STEP 2093 H=2093:00 +CPU= 23.624
  16:16:33 STEP 2094 H=2094:00 +CPU= 37.344
  16:16:59 STEP 2095 H=2095:00 +CPU= 26.076
  16:17:27 STEP 2096 H=2096:00 +CPU= 26.917
  16:18:04 STEP 2097 H=2097:00 +CPU= 35.527
  16:18:29 STEP 2098 H=2098:00 +CPU= 24.387
  16:18:53 STEP 2099 H=2099:00 +CPU= 23.208
  16:19:34 STEP 2100 H=2100:00 +CPU= 39.427
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002100
  16:19:58 STEP 2101 H=2101:00 +CPU= 23.703
  16:20:26 STEP 2102 H=2102:00 +CPU= 26.528
  16:20:59 STEP 2103 H=2103:00 +CPU= 32.801
  16:21:25 STEP 2104 H=2104:00 +CPU= 25.252
  16:21:51 STEP 2105 H=2105:00 +CPU= 25.290
  16:22:33 STEP 2106 H=2106:00 +CPU= 40.891
  16:22:55 STEP 2107 H=2107:00 +CPU= 21.651
  16:23:23 STEP 2108 H=2108:00 +CPU= 26.809
  16:24:04 STEP 2109 H=2109:00 +CPU= 40.307
  16:24:30 STEP 2110 H=2110:00 +CPU= 24.581
  16:24:52 STEP 2111 H=2111:00 +CPU= 21.940
  16:25:38 STEP 2112 H=2112:00 +CPU= 44.451
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_87.zip
Uploading the intermediate file: upload_file_87.zip
  16:26:04 STEP 2113 H=2113:00 +CPU= 24.978
  16:26:29 STEP 2114 H=2114:00 +CPU= 24.359
  16:27:11 STEP 2115 H=2115:00 +CPU= 40.066
  16:27:35 STEP 2116 H=2116:00 +CPU= 24.080
  16:28:02 STEP 2117 H=2117:00 +CPU= 26.040
  16:28:43 STEP 2118 H=2118:00 +CPU= 40.380
  16:29:05 STEP 2119 H=2119:00 +CPU= 21.211
  16:29:32 STEP 2120 H=2120:00 +CPU= 26.277
  16:30:12 STEP 2121 H=2121:00 +CPU= 39.721
  16:30:39 STEP 2122 H=2122:00 +CPU= 26.932
  16:31:04 STEP 2123 H=2123:00 +CPU= 23.515
  16:31:46 STEP 2124 H=2124:00 +CPU= 41.119
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002124
  16:32:09 STEP 2125 H=2125:00 +CPU= 22.483
  16:32:37 STEP 2126 H=2126:00 +CPU= 26.781
  16:33:18 STEP 2127 H=2127:00 +CPU= 40.247
  16:33:41 STEP 2128 H=2128:00 +CPU= 22.609
  16:34:06 STEP 2129 H=2129:00 +CPU= 23.238
  16:34:46 STEP 2130 H=2130:00 +CPU= 39.273
  16:35:11 STEP 2131 H=2131:00 +CPU= 23.929
  16:35:37 STEP 2132 H=2132:00 +CPU= 25.670
  16:36:18 STEP 2133 H=2133:00 +CPU= 40.325
  16:36:45 STEP 2134 H=2134:00 +CPU= 26.848
  16:37:13 STEP 2135 H=2135:00 +CPU= 26.555
  16:37:52 STEP 2136 H=2136:00 +CPU= 39.023
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071_88.zip
Uploading the intermediate file: upload_file_88.zip
  16:38:13 STEP 2137 H=2137:00 +CPU= 21.104
  16:38:36 STEP 2138 H=2138:00 +CPU= 22.204
  16:39:16 STEP 2139 H=2139:00 +CPU= 39.848
  16:39:44 STEP 2140 H=2140:00 +CPU= 26.802
  16:40:10 STEP 2141 H=2141:00 +CPU= 26.006
  16:40:49 STEP 2142 H=2142:00 +CPU= 38.252
  16:41:11 STEP 2143 H=2143:00 +CPU= 21.165
  16:41:35 STEP 2144 H=2144:00 +CPU= 23.405
  16:42:16 STEP 2145 H=2145:00 +CPU= 40.335
  16:42:44 STEP 2146 H=2146:00 +CPU= 26.557
  16:43:11 STEP 2147 H=2147:00 +CPU= 25.053
  16:43:52 STEP 2148 H=2148:00 +CPU= 39.499
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002148
  16:44:17 STEP 2149 H=2149:00 +CPU= 23.923
  16:44:42 STEP 2150 H=2150:00 +CPU= 23.237
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.014] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4035MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.014] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.014] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.014] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.014] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.014] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4035 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_ifs_rrtm:gas_optics 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164449:1674866689:7086.015] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  radiation_ifs_rrtm:planck_function 
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164450:1674866690:7087.015] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:naturerick:1:1:3466:3466] [20230127:164450:1674866690:7087.015] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000018BBA47  radiation_ifs_rrt         377  radiation_ifs_rrtm.F90
oifs_43r3_model.e  0000000001862293  radiation_interfa         311  radiation_interface.F90
oifs_43r3_model.e  00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  000000000129A415  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0926
(argv4) batchid: 970
(argv5) wuid: 12187570
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0926
(argv4) batchid: 970
(argv5) wuid: 12187570
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_06eef93e84eaba0d37d7b5ba8532f13a to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0926_2001050100_123_970_12187570.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0926_2001050100_123_970_12187570.zip
ic_ancil_file: ic_ancil_12187570
ifsdata_file: ifsdata_12187570
climate_data_file: clim_data_12187570
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1a982f0721d6285eb60a6a278841687e to: /var/lib/boinc-client/slots/1/ic_ancil_12187570.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12187570.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_df5a59b43064d0dd86b10fd669595888 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12187570.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12187570.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12187570.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12187570.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12187570.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12187570.xml
-- Model is restarting --
Adjusting last_iter, 2151, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12187570.xml
last_cpu_time: 64184
upload_file_number: 89
last_iter: 2135
last_upload: 7689600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071
The child process has been launched with process id: 3382
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.014] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.015] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.015] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.015] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.016] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.016] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.017] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.017] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.018] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.018] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.019] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.019] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.019] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.022] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.022] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.022] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.024] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.024] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.024] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.026] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.026] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.026] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.027] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.027] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.029] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.029] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.029] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:002318:1675066998:0.031] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:29:32 STEP 2136 H=2136:00 +CPU=184.786
  00:32:02 STEP 2137 H=2137:00 +CPU=104.115
  00:34:48 STEP 2138 H=2138:00 +CPU=108.836
  00:40:58 STEP 2139 H=2139:00 +CPU=164.128
  00:45:01 STEP 2140 H=2140:00 +CPU=107.353
  00:48:53 STEP 2141 H=2141:00 +CPU=103.710
  00:55:01 STEP 2142 H=2142:00 +CPU=166.887
  00:58:45 STEP 2143 H=2143:00 +CPU=104.299
  01:02:33 STEP 2144 H=2144:00 +CPU=107.207
  01:08:31 STEP 2145 H=2145:00 +CPU=164.976
  01:12:31 STEP 2146 H=2146:00 +CPU=107.186
  01:16:28 STEP 2147 H=2147:00 +CPU=103.798
  01:20:29 STEP 2148 H=2148:00 +CPU=185.870
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002148
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.431] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.431] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.431] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.431] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
double free or corruption (top)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.432] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012219:1675070539:3541.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012220:1675070540:3542.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:naturerick:1:1:3382:3382] [20230130:012220:1675070540:3542.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001016488  cloudsc_                  709  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F1A790  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0926
(argv4) batchid: 970
(argv5) wuid: 12187570
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187570
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_06eef93e84eaba0d37d7b5ba8532f13a to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0926_2001050100_123_970_12187570.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0926_2001050100_123_970_12187570.zip
ic_ancil_file: ic_ancil_12187570
ifsdata_file: ifsdata_12187570
climate_data_file: clim_data_12187570
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1a982f0721d6285eb60a6a278841687e to: /var/lib/boinc-client/slots/1/ic_ancil_12187570.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12187570.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_df5a59b43064d0dd86b10fd669595888 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12187570.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12187570.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12187570.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12187570.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12187570.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12187570.xml
-- Model is restarting --
Adjusting last_iter, 2149, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12187570.xml
last_cpu_time: 65983
upload_file_number: 89
last_iter: 2135
last_upload: 7689600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0926_2001050100_123_970_12187570_2_r16952071
The child process has been launched with process id: 3781
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.003] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.009] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3781:3781] [20230130:103220:1675103540:0.010] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:38:11 STEP 2136 H=2136:00 +CPU=181.873
  10:40:38 STEP 2137 H=2137:00 +CPU=103.373
  10:43:08 STEP 2138 H=2138:00 +CPU=105.952
  10:47:01 STEP 2139 H=2139:00 +CPU=160.087
  10:49:34 STEP 2140 H=2140:00 +CPU=106.220
  10:51:56 STEP 2141 H=2141:00 +CPU=104.419
  10:54:54 STEP 2142 H=2142:00 +CPU=168.456
  10:56:35 STEP 2143 H=2143:00 +CPU= 96.613
  10:58:20 STEP 2144 H=2144:00 +CPU=101.413
  11:01:14 STEP 2145 H=2145:00 +CPU=167.699
  11:03:02 STEP 2146 H=2146:00 +CPU=104.287
  11:04:45 STEP 2147 H=2147:00 +CPU= 99.080
  11:08:02 STEP 2148 H=2148:00 +CPU=190.331
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002148
  11:09:48 STEP 2149 H=2149:00 +CPU=100.384
  11:11:48 STEP 2150 H=2150:00 +CPU=115.241
  11:14:39 STEP 2151 H=2151:00 +CPU=164.173
  11:16:32 STEP 2152 H=2152:00 +CPU=108.421
  11:18:26 STEP 2153 H=2153:00 +CPU=109.855
  11:21:32 STEP 2154 H=2154:00 +CPU=174.410
  11:23:11 STEP 2155 H=2155:00 +CPU= 95.662
  11:25:06 STEP 2156 H=2156:00 +CPU=111.112
  11:29:13 STEP 2157 H=2157:00 +CPU=164.925
  11:33:11 STEP 2158 H=2158:00 +CPU=108.821
  11:37:15 STEP 2159 H=2159:00 +CPU=107.361
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:43:39 1506016 22302803 oifs_43r3_ps_0926_2001050100_123_970_12187570_2 6,566,400 53,798 0.0082
09 May 2023 16:42:35 1506016 22302803 oifs_43r3_ps_0926_2001050100_123_970_12187570_2 5,529,600 44,952 0.0081
09 May 2023 16:42:31 1506016 22302803 oifs_43r3_ps_0926_2001050100_123_970_12187570_2 4,665,600 38,689 0.0083
09 May 2023 16:42:25 1506016 22302803 oifs_43r3_ps_0926_2001050100_123_970_12187570_2 3,801,600 32,470 0.0085
09 May 2023 16:39:58 1506016 22302803 oifs_43r3_ps_0926_2001050100_123_970_12187570_2 1,987,200 16,519 0.0083
09 May 2023 16:39:51 1506016 22302803 oifs_43r3_ps_0926_2001050100_123_970_12187570_2 1,123,200 9,796 0.0087


©2024 climateprediction.net