climateprediction.net home page
Task 22261460

Task 22261460

Name oifs_43r3_ps_0056_1991050100_123_960_12176700_0
Workunit 12176700
Created 21 Dec 2022, 20:48:37 UTC
Sent 20 Jan 2023, 18:18:50 UTC
Report deadline 21 Mar 2023, 18:18:50 UTC
Received 25 Jan 2023, 12:38:26 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1509861
Run time 7 hours 38 min 9 sec
CPU time 7 hours 11 min 14 sec
Validate state Invalid
Credit 1,209.60
Device peak FLOPS 5.68 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.12 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,888.02 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:223729:1674509849:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:39:18 STEP  888 H= 888:00 +CPU= 37.416
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:40:09 STEP  889 H= 889:00 +CPU= 18.938
  22:40:28 STEP  890 H= 890:00 +CPU= 17.546
  22:40:56 STEP  891 H= 891:00 +CPU= 27.297
  22:41:15 STEP  892 H= 892:00 +CPU= 18.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:41:47 STEP  893 H= 893:00 +CPU= 18.857
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:42:31 STEP  894 H= 894:00 +CPU= 30.163
  22:42:51 STEP  895 H= 895:00 +CPU= 18.056
  22:43:11 STEP  896 H= 896:00 +CPU= 18.549
  22:43:40 STEP  897 H= 897:00 +CPU= 27.831
  22:43:59 STEP  898 H= 898:00 +CPU= 17.639
  22:44:16 STEP  899 H= 899:00 +CPU= 17.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:45:23 STEP  900 H= 900:00 +CPU= 32.106
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000900
  22:45:44 STEP  901 H= 901:00 +CPU= 18.741
  22:46:03 STEP  902 H= 902:00 +CPU= 18.684
  22:46:33 STEP  903 H= 903:00 +CPU= 28.174
  22:46:53 STEP  904 H= 904:00 +CPU= 18.535
  22:47:12 STEP  905 H= 905:00 +CPU= 17.928
  22:47:43 STEP  906 H= 906:00 +CPU= 28.976
  22:48:02 STEP  907 H= 907:00 +CPU= 18.052
  22:48:22 STEP  908 H= 908:00 +CPU= 18.441
  22:48:52 STEP  909 H= 909:00 +CPU= 28.191
  22:49:11 STEP  910 H= 910:00 +CPU= 18.604
  22:49:31 STEP  911 H= 911:00 +CPU= 18.587
  22:50:05 STEP  912 H= 912:00 +CPU= 32.614
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_37.zip
Uploading the intermediate file: upload_file_37.zip
  22:50:23 STEP  913 H= 913:00 +CPU= 17.161
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:50:55 STEP  914 H= 914:00 +CPU= 19.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:51:38 STEP  915 H= 915:00 +CPU= 29.893
  22:51:59 STEP  916 H= 916:00 +CPU= 19.214
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:52:29 STEP  917 H= 917:00 +CPU= 18.479
  22:53:01 STEP  918 H= 918:00 +CPU= 29.724
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:45 STEP  919 H= 919:00 +CPU= 20.775
  22:54:05 STEP  920 H= 920:00 +CPU= 18.867
  22:54:36 STEP  921 H= 921:00 +CPU= 28.509
  22:54:54 STEP  922 H= 922:00 +CPU= 17.668
  22:55:13 STEP  923 H= 923:00 +CPU= 18.013
  22:55:44 STEP  924 H= 924:00 +CPU= 30.367
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000924
  22:56:03 STEP  925 H= 925:00 +CPU= 18.177
  22:56:23 STEP  926 H= 926:00 +CPU= 18.324
  22:56:52 STEP  927 H= 927:00 +CPU= 27.626
  22:57:12 STEP  928 H= 928:00 +CPU= 18.050
  22:57:31 STEP  929 H= 929:00 +CPU= 17.968
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:58:16 STEP  930 H= 930:00 +CPU= 31.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:58:47 STEP  931 H= 931:00 +CPU= 18.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:59:21 STEP  932 H= 932:00 +CPU= 20.229
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:04 STEP  933 H= 933:00 +CPU= 29.326
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:45 STEP  934 H= 934:00 +CPU= 18.554
  23:01:04 STEP  935 H= 935:00 +CPU= 18.165
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:01:52 STEP  936 H= 936:00 +CPU= 35.251
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_38.zip
Uploading the intermediate file: upload_file_38.zip
  23:02:11 STEP  937 H= 937:00 +CPU= 17.724
  23:02:31 STEP  938 H= 938:00 +CPU= 18.203
  23:03:01 STEP  939 H= 939:00 +CPU= 27.728
  23:03:19 STEP  940 H= 940:00 +CPU= 17.542
  23:03:36 STEP  941 H= 941:00 +CPU= 17.125
  23:04:05 STEP  942 H= 942:00 +CPU= 27.652
  23:04:24 STEP  943 H= 943:00 +CPU= 17.797
  23:04:44 STEP  944 H= 944:00 +CPU= 19.235
  23:05:15 STEP  945 H= 945:00 +CPU= 28.670
  23:05:35 STEP  946 H= 946:00 +CPU= 18.468
  23:05:55 STEP  947 H= 947:00 +CPU= 18.095
  23:06:29 STEP  948 H= 948:00 +CPU= 31.774
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000948
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:07:12 STEP  949 H= 949:00 +CPU= 19.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:08:07 STEP  950 H= 950:00 +CPU= 21.055
  23:08:37 STEP  951 H= 951:00 +CPU= 28.636
  23:08:57 STEP  952 H= 952:00 +CPU= 18.353
  23:09:17 STEP  953 H= 953:00 +CPU= 18.391
  23:09:48 STEP  954 H= 954:00 +CPU= 29.546
  23:10:07 STEP  955 H= 955:00 +CPU= 18.101
  23:10:28 STEP  956 H= 956:00 +CPU= 18.829
  23:10:59 STEP  957 H= 957:00 +CPU= 29.068
  23:11:19 STEP  958 H= 958:00 +CPU= 18.649
  23:11:39 STEP  959 H= 959:00 +CPU= 18.517
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:12:27 STEP  960 H= 960:00 +CPU= 34.463
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_39.zip
Uploading the intermediate file: upload_file_39.zip
  23:12:47 STEP  961 H= 961:00 +CPU= 18.849
  23:13:07 STEP  962 H= 962:00 +CPU= 18.918
  23:13:37 STEP  963 H= 963:00 +CPU= 28.179
  23:13:57 STEP  964 H= 964:00 +CPU= 18.314
  23:14:16 STEP  965 H= 965:00 +CPU= 17.687
  23:14:47 STEP  966 H= 966:00 +CPU= 28.815
  23:15:06 STEP  967 H= 967:00 +CPU= 18.400
  23:15:25 STEP  968 H= 968:00 +CPU= 18.083
  23:15:55 STEP  969 H= 969:00 +CPU= 28.269
  23:16:14 STEP  970 H= 970:00 +CPU= 18.683
  23:16:33 STEP  971 H= 971:00 +CPU= 17.860
  23:17:05 STEP  972 H= 972:00 +CPU= 30.554
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000972
  23:17:24 STEP  973 H= 973:00 +CPU= 17.884
  23:17:43 STEP  974 H= 974:00 +CPU= 18.304
  23:18:13 STEP  975 H= 975:00 +CPU= 28.339
  23:18:32 STEP  976 H= 976:00 +CPU= 18.422
  23:18:51 STEP  977 H= 977:00 +CPU= 17.746
  23:19:21 STEP  978 H= 978:00 +CPU= 28.687
  23:19:39 STEP  979 H= 979:00 +CPU= 17.678
  23:19:58 STEP  980 H= 980:00 +CPU= 18.412
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:21:12 STEP  981 H= 981:00 +CPU= 30.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:22:54 STEP  982 H= 982:00 +CPU= 19.873
  23:23:14 STEP  983 H= 983:00 +CPU= 18.181
  23:23:49 STEP  984 H= 984:00 +CPU= 32.651
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_40.zip
Uploading the intermediate file: upload_file_40.zip
  23:24:08 STEP  985 H= 985:00 +CPU= 17.678
  23:24:28 STEP  986 H= 986:00 +CPU= 18.150
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:21 STEP  987 H= 987:00 +CPU= 29.815
  23:25:42 STEP  988 H= 988:00 +CPU= 20.110
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:26:13 STEP  989 H= 989:00 +CPU= 17.989
  23:26:43 STEP  990 H= 990:00 +CPU= 28.785
  23:27:02 STEP  991 H= 991:00 +CPU= 17.722
  23:27:23 STEP  992 H= 992:00 +CPU= 19.019
  23:27:53 STEP  993 H= 993:00 +CPU= 28.585
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:28:26 STEP  994 H= 994:00 +CPU= 20.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:29:54 STEP  995 H= 995:00 +CPU= 20.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:30:38 STEP  996 H= 996:00 +CPU= 31.123
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000996
  23:30:57 STEP  997 H= 997:00 +CPU= 17.675
  23:31:17 STEP  998 H= 998:00 +CPU= 18.886
  23:31:47 STEP  999 H= 999:00 +CPU= 28.359
  23:32:07 STEP 1000 H=1000:00 +CPU= 18.996
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:32:49 STEP 1001 H=1001:00 +CPU= 20.027
  23:33:19 STEP 1002 H=1002:00 +CPU= 28.167
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3547MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3547 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CPG_DRV 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-CPG                  (1025) 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              CPG 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CPG_GP 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                GPCTY 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VERINT 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233322:1674513202:3352.670] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VERINT_DGEMM_2 
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233323:1674513203:3353.670] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2119:2119] [20230123:233323:1674513203:3353.671] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001711658  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000A3E48D  verint_                   168  verint.F90
oifs_43r3_model.e  0000000000C6255C  gpcty_                    278  gpcty.F90
oifs_43r3_model.e  0000000000C393EA  cpg_gp_                   788  cpg_gp.F90
oifs_43r3_model.e  0000000000C2E72E  cpg_                      627  cpg.F90
oifs_43r3_model.e  0000000000C2C638  cpg_drv_                  351  cpg_drv.F90
oifs_43r3_model.e  0000000000BF17C8  gp_model_                 419  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0056
(argv4) batchid: 960
(argv5) wuid: 12176700
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_07c619bc40a95c5326a942a5be913f56 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0056_1991050100_123_960_12176700.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0056_1991050100_123_960_12176700.zip
ic_ancil_file: ic_ancil_12176700
ifsdata_file: ifsdata_12176700
climate_data_file: clim_data_12176700
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8d32566c8244c3ab6d6c331a922f27b4 to: /var/lib/boinc-client/slots/0/ic_ancil_12176700.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12176700.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eaa1b009539d1fa9bae812c9228497ec to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12176700.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12176700.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12176700.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12176700.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12176700.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12176700.xml
-- Model is restarting --
Adjusting last_iter, 1002, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12176700.xml
last_cpu_time: 23364
upload_file_number: 41
last_iter: 983
last_upload: 3542400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028
The child process has been launched with process id: 2084
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:222615:1674595575:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:27:26 STEP  984 H= 984:00 +CPU= 32.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:56 STEP  985 H= 985:00 +CPU= 18.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:29:28 STEP  986 H= 986:00 +CPU= 19.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:30:11 STEP  987 H= 987:00 +CPU= 29.698
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:30:45 STEP  988 H= 988:00 +CPU= 20.413
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:31:15 STEP  989 H= 989:00 +CPU= 18.876
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:33:12 STEP  990 H= 990:00 +CPU= 31.286
  22:33:32 STEP  991 H= 991:00 +CPU= 19.032
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:34:04 STEP  992 H= 992:00 +CPU= 19.884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:34:58 STEP  993 H= 993:00 +CPU= 30.251
  22:35:19 STEP  994 H= 994:00 +CPU= 19.299
  22:35:39 STEP  995 H= 995:00 +CPU= 18.744
  22:36:15 STEP  996 H= 996:00 +CPU= 33.752
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000996
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:36:46 STEP  997 H= 997:00 +CPU= 18.489
  22:37:04 STEP  998 H= 998:00 +CPU= 17.779
  22:37:34 STEP  999 H= 999:00 +CPU= 28.238
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:38:07 STEP 1000 H=1000:00 +CPU= 20.694
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:38:50 STEP 1001 H=1001:00 +CPU= 19.481
  22:39:23 STEP 1002 H=1002:00 +CPU= 30.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:39:54 STEP 1003 H=1003:00 +CPU= 19.151
  22:40:16 STEP 1004 H=1004:00 +CPU= 19.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:40:59 STEP 1005 H=1005:00 +CPU= 30.267
  22:41:20 STEP 1006 H=1006:00 +CPU= 19.835
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:42:04 STEP 1007 H=1007:00 +CPU= 20.529
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:42:52 STEP 1008 H=1008:00 +CPU= 34.103
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_41.zip
Uploading the intermediate file: upload_file_41.zip
  22:43:13 STEP 1009 H=1009:00 +CPU= 19.328
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:44:05 STEP 1010 H=1010:00 +CPU= 19.286
  22:44:35 STEP 1011 H=1011:00 +CPU= 28.334
  22:44:55 STEP 1012 H=1012:00 +CPU= 18.380
  22:45:15 STEP 1013 H=1013:00 +CPU= 18.150
  22:45:46 STEP 1014 H=1014:00 +CPU= 29.023
  22:46:05 STEP 1015 H=1015:00 +CPU= 18.090
  22:46:25 STEP 1016 H=1016:00 +CPU= 18.658
  22:46:56 STEP 1017 H=1017:00 +CPU= 28.313
  22:47:15 STEP 1018 H=1018:00 +CPU= 18.511
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:48:18 STEP 1019 H=1019:00 +CPU= 19.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:49:36 STEP 1020 H=1020:00 +CPU= 33.078
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001020
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:50:08 STEP 1021 H=1021:00 +CPU= 19.664
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:50:40 STEP 1022 H=1022:00 +CPU= 19.212
  22:51:12 STEP 1023 H=1023:00 +CPU= 30.011
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:52:06 STEP 1024 H=1024:00 +CPU= 20.352
  22:52:26 STEP 1025 H=1025:00 +CPU= 19.057
  22:52:59 STEP 1026 H=1026:00 +CPU= 30.077
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:30 STEP 1027 H=1027:00 +CPU= 19.087
  22:53:51 STEP 1028 H=1028:00 +CPU= 19.397
  22:54:22 STEP 1029 H=1029:00 +CPU= 28.977
  22:54:43 STEP 1030 H=1030:00 +CPU= 19.215
  22:55:04 STEP 1031 H=1031:00 +CPU= 19.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:55:53 STEP 1032 H=1032:00 +CPU= 35.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_42.zip
Uploading the intermediate file: upload_file_42.zip
  22:56:25 STEP 1033 H=1033:00 +CPU= 19.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:09 STEP 1034 H=1034:00 +CPU= 19.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:53 STEP 1035 H=1035:00 +CPU= 30.235
  22:58:14 STEP 1036 H=1036:00 +CPU= 20.010
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:58:56 STEP 1037 H=1037:00 +CPU= 18.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:59:52 STEP 1038 H=1038:00 +CPU= 31.452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:24 STEP 1039 H=1039:00 +CPU= 19.532
  23:00:45 STEP 1040 H=1040:00 +CPU= 20.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:03:10 STEP 1041 H=1041:00 +CPU= 29.504
  23:03:28 STEP 1042 H=1042:00 +CPU= 17.853
  23:03:47 STEP 1043 H=1043:00 +CPU= 17.459
  23:04:18 STEP 1044 H=1044:00 +CPU= 30.362
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001044
  23:04:38 STEP 1045 H=1045:00 +CPU= 17.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:05:21 STEP 1046 H=1046:00 +CPU= 19.067
  23:05:51 STEP 1047 H=1047:00 +CPU= 27.552
  23:06:09 STEP 1048 H=1048:00 +CPU= 17.788
  23:06:27 STEP 1049 H=1049:00 +CPU= 17.091
  23:06:56 STEP 1050 H=1050:00 +CPU= 27.960
  23:07:14 STEP 1051 H=1051:00 +CPU= 17.338
  23:07:33 STEP 1052 H=1052:00 +CPU= 18.006
  23:08:02 STEP 1053 H=1053:00 +CPU= 27.247
  23:08:20 STEP 1054 H=1054:00 +CPU= 17.864
  23:08:39 STEP 1055 H=1055:00 +CPU= 18.006
  23:09:13 STEP 1056 H=1056:00 +CPU= 32.259
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_43.zip
Uploading the intermediate file: upload_file_43.zip
  23:09:32 STEP 1057 H=1057:00 +CPU= 18.256
  23:09:51 STEP 1058 H=1058:00 +CPU= 18.204
  23:10:20 STEP 1059 H=1059:00 +CPU= 27.752
  23:10:38 STEP 1060 H=1060:00 +CPU= 17.756
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.255] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2550MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.258] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.258] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.258] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.258] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.258] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2598 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.258] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          WAMODEL 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231056:1674598256:2681.282] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           >OMP-WAMODEL 4            (1439) 
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231057:1674598257:2682.282] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2084:2084] [20230124:231057:1674598257:2682.282] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DAFCE2  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000168DC46  wamodel_                 1115  wamodel.F
oifs_43r3_model.e  0000000000872295  wavemdl_                  584  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0056
(argv4) batchid: 960
(argv5) wuid: 12176700
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_07c619bc40a95c5326a942a5be913f56 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0056_1991050100_123_960_12176700.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0056_1991050100_123_960_12176700.zip
ic_ancil_file: ic_ancil_12176700
ifsdata_file: ifsdata_12176700
climate_data_file: clim_data_12176700
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8d32566c8244c3ab6d6c331a922f27b4 to: /var/lib/boinc-client/slots/0/ic_ancil_12176700.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12176700.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eaa1b009539d1fa9bae812c9228497ec to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12176700.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12176700.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12176700.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12176700.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12176700.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12176700.xml
-- Model is restarting --
Adjusting last_iter, 1061, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12176700.xml
last_cpu_time: 25128
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028
The child process has been launched with process id: 2070
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2070:2070] [20230125:132144:1674649304:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:23:09 STEP 1056 H=1056:00 +CPU= 34.512
  13:23:26 STEP 1057 H=1057:00 +CPU= 17.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:24:50 STEP 1058 H=1058:00 +CPU= 19.907
  13:25:18 STEP 1059 H=1059:00 +CPU= 26.984
  13:25:36 STEP 1060 H=1060:00 +CPU= 17.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:26:08 STEP 1061 H=1061:00 +CPU= 18.991
  13:26:38 STEP 1062 H=1062:00 +CPU= 28.917
  13:26:56 STEP 1063 H=1063:00 +CPU= 17.203
  13:27:14 STEP 1064 H=1064:00 +CPU= 17.586
  13:27:42 STEP 1065 H=1065:00 +CPU= 27.133
  13:28:00 STEP 1066 H=1066:00 +CPU= 17.601
  13:28:18 STEP 1067 H=1067:00 +CPU= 17.136
  13:28:49 STEP 1068 H=1068:00 +CPU= 29.863
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001068
  13:29:07 STEP 1069 H=1069:00 +CPU= 17.144
  13:29:25 STEP 1070 H=1070:00 +CPU= 17.647
  13:29:53 STEP 1071 H=1071:00 +CPU= 27.062
  13:30:12 STEP 1072 H=1072:00 +CPU= 18.111
  13:30:31 STEP 1073 H=1073:00 +CPU= 17.817
  13:31:01 STEP 1074 H=1074:00 +CPU= 28.312
  13:31:19 STEP 1075 H=1075:00 +CPU= 17.426
  13:31:38 STEP 1076 H=1076:00 +CPU= 17.983
  13:32:07 STEP 1077 H=1077:00 +CPU= 27.688
  13:32:26 STEP 1078 H=1078:00 +CPU= 18.310
  13:32:45 STEP 1079 H=1079:00 +CPU= 18.028
  13:33:19 STEP 1080 H=1080:00 +CPU= 32.216
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12176700/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0056_1991050100_123_960_12176700_0_r936815028_44.zip
Uploading the intermediate file: upload_file_44.zip
  13:33:37 STEP 1081 H=1081:00 +CPU= 17.459
  13:33:55 STEP 1082 H=1082:00 +CPU= 17.864
  13:34:24 STEP 1083 H=1083:00 +CPU= 27.541
  13:34:43 STEP 1084 H=1084:00 +CPU= 17.816
  13:35:01 STEP 1085 H=1085:00 +CPU= 17.485
  13:35:31 STEP 1086 H=1086:00 +CPU= 28.350
  13:35:49 STEP 1087 H=1087:00 +CPU= 17.312
  13:36:07 STEP 1088 H=1088:00 +CPU= 17.870
  13:36:36 STEP 1089 H=1089:00 +CPU= 27.341
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:31:34 1509861 22261460 oifs_43r3_ps_0056_1991050100_123_960_12176700_0 864,000 5,275 0.0061


©2024 climateprediction.net