Name | oifs_43r3_ps_0409_2002050100_123_971_12188053_0 |
Workunit | 12188053 |
Created | 23 Dec 2022, 2:39:35 UTC |
Sent | 7 Jan 2023, 5:40:13 UTC |
Report deadline | 6 Feb 2023, 5:40:13 UTC |
Received | 10 Jan 2023, 10:38:14 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1491147 |
Run time | 15 hours 3 min 39 sec |
CPU time | 12 hours 33 min 21 sec |
Validate state | Invalid |
Credit | 1,209.60 |
Device peak FLOPS | 4.56 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,484.85 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,483.75 MB |
Stderr | <core_client_version>7.16.16</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> s_12188053/ICMUAhq0f+000876 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516_36.zip Uploading the intermediate file: upload_file_36.zip 18:47:02 STEP 889 H= 889:00 +CPU= 37.658 18:47:37 STEP 890 H= 890:00 +CPU= 34.082 18:48:31 STEP 891 H= 891:00 +CPU= 52.682 18:49:11 STEP 892 H= 892:00 +CPU= 39.676 18:49:49 STEP 893 H= 893:00 +CPU= 38.173 18:50:42 STEP 894 H= 894:00 +CPU= 51.955 18:51:15 STEP 895 H= 895:00 +CPU= 32.956 18:51:50 STEP 896 H= 896:00 +CPU= 33.616 18:52:40 STEP 897 H= 897:00 +CPU= 49.794 18:53:15 STEP 898 H= 898:00 +CPU= 34.571 18:53:48 STEP 899 H= 899:00 +CPU= 32.829 18:54:53 STEP 900 H= 900:00 +CPU= 56.076 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000900 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000900 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000900 18:55:26 STEP 901 H= 901:00 +CPU= 32.993 18:55:57 STEP 902 H= 902:00 +CPU= 29.566 19:02:15 STEP 903 H= 903:00 +CPU= 50.553 19:03:54 STEP 904 H= 904:00 +CPU= 31.571 19:07:50 STEP 905 H= 905:00 +CPU= 28.395 19:10:13 STEP 906 H= 906:00 +CPU= 43.072 19:10:39 STEP 907 H= 907:00 +CPU= 25.265 19:13:10 STEP 908 H= 908:00 +CPU= 30.812 19:17:29 STEP 909 H= 909:00 +CPU= 50.060 19:18:01 STEP 910 H= 910:00 +CPU= 31.307 19:20:04 STEP 911 H= 911:00 +CPU= 30.652 19:22:31 STEP 912 H= 912:00 +CPU= 52.510 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000912 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000912 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000900 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516_37.zip Uploading the intermediate file: upload_file_37.zip 19:23:59 STEP 913 H= 913:00 +CPU= 30.367 19:25:17 STEP 914 H= 914:00 +CPU= 25.428 19:27:18 STEP 915 H= 915:00 +CPU= 43.361 19:29:26 STEP 916 H= 916:00 +CPU= 29.378 19:30:04 STEP 917 H= 917:00 +CPU= 28.096 19:30:49 STEP 918 H= 918:00 +CPU= 44.930 19:31:18 STEP 919 H= 919:00 +CPU= 28.277 19:32:14 STEP 920 H= 920:00 +CPU= 25.717 19:33:00 STEP 921 H= 921:00 +CPU= 39.890 19:33:31 STEP 922 H= 922:00 +CPU= 30.837 19:35:03 STEP 923 H= 923:00 +CPU= 26.593 19:36:28 STEP 924 H= 924:00 +CPU= 51.748 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000924 19:37:00 STEP 925 H= 925:00 +CPU= 29.657 19:37:37 STEP 926 H= 926:00 +CPU= 30.633 19:39:41 STEP 927 H= 927:00 +CPU= 42.942 19:40:24 STEP 928 H= 928:00 +CPU= 32.180 19:40:55 STEP 929 H= 929:00 +CPU= 30.648 19:41:44 STEP 930 H= 930:00 +CPU= 48.407 19:42:15 STEP 931 H= 931:00 +CPU= 31.330 19:42:46 STEP 932 H= 932:00 +CPU= 30.784 19:43:34 STEP 933 H= 933:00 +CPU= 46.938 19:44:04 STEP 934 H= 934:00 +CPU= 30.039 19:44:37 STEP 935 H= 935:00 +CPU= 31.568 19:45:41 STEP 936 H= 936:00 +CPU= 56.158 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000924 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516_38.zip Uploading the intermediate file: upload_file_38.zip 19:46:12 STEP 937 H= 937:00 +CPU= 30.702 19:47:10 STEP 938 H= 938:00 +CPU= 30.050 19:48:04 STEP 939 H= 939:00 +CPU= 45.659 19:48:33 STEP 940 H= 940:00 +CPU= 28.332 19:49:11 STEP 941 H= 941:00 +CPU= 28.843 19:49:58 STEP 942 H= 942:00 +CPU= 46.536 19:50:38 STEP 943 H= 943:00 +CPU= 28.676 19:51:32 STEP 944 H= 944:00 +CPU= 32.269 19:52:44 STEP 945 H= 945:00 +CPU= 46.421 19:54:24 STEP 946 H= 946:00 +CPU= 27.726 19:54:56 STEP 947 H= 947:00 +CPU= 27.351 19:55:48 STEP 948 H= 948:00 +CPU= 49.802 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000948 19:56:16 STEP 949 H= 949:00 +CPU= 28.196 19:57:37 STEP 950 H= 950:00 +CPU= 27.481 19:58:23 STEP 951 H= 951:00 +CPU= 45.333 19:58:59 STEP 952 H= 952:00 +CPU= 27.646 19:59:29 STEP 953 H= 953:00 +CPU= 30.225 20:00:26 STEP 954 H= 954:00 +CPU= 47.876 20:00:55 STEP 955 H= 955:00 +CPU= 26.271 20:01:55 STEP 956 H= 956:00 +CPU= 29.886 20:03:29 STEP 957 H= 957:00 +CPU= 45.446 20:06:02 STEP 958 H= 958:00 +CPU= 29.925 20:06:38 STEP 959 H= 959:00 +CPU= 28.198 20:10:07 STEP 960 H= 960:00 +CPU= 50.616 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000960 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000960 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000948 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000948 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000948 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516_39.zip Uploading the intermediate file: upload_file_39.zip 20:12:48 STEP 961 H= 961:00 +CPU= 29.552 20:14:57 STEP 962 H= 962:00 +CPU= 32.407 20:17:49 STEP 963 H= 963:00 +CPU= 43.861 20:18:19 STEP 964 H= 964:00 +CPU= 28.511 20:20:21 STEP 965 H= 965:00 +CPU= 29.789 20:21:09 STEP 966 H= 966:00 +CPU= 46.257 20:21:40 STEP 967 H= 967:00 +CPU= 30.390 20:22:11 STEP 968 H= 968:00 +CPU= 31.163 20:23:11 STEP 969 H= 969:00 +CPU= 44.970 20:23:57 STEP 970 H= 970:00 +CPU= 28.206 20:24:22 STEP 971 H= 971:00 +CPU= 23.525 20:25:33 STEP 972 H= 972:00 +CPU= 47.371 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000972 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000972 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000972 20:26:02 STEP 973 H= 973:00 +CPU= 28.977 20:27:17 STEP 974 H= 974:00 +CPU= 27.388 20:28:03 STEP 975 H= 975:00 +CPU= 42.745 20:29:17 STEP 976 H= 976:00 +CPU= 28.295 20:30:55 STEP 977 H= 977:00 +CPU= 26.484 20:31:41 STEP 978 H= 978:00 +CPU= 43.967 20:34:10 STEP 979 H= 979:00 +CPU= 30.261 20:34:49 STEP 980 H= 980:00 +CPU= 29.111 20:35:43 STEP 981 H= 981:00 +CPU= 45.317 20:36:38 STEP 982 H= 982:00 +CPU= 31.958 20:37:15 STEP 983 H= 983:00 +CPU= 29.383 20:39:04 STEP 984 H= 984:00 +CPU= 49.969 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000984 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000984 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000972 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516_40.zip Uploading the intermediate file: upload_file_40.zip 20:39:32 STEP 985 H= 985:00 +CPU= 27.322 20:40:01 STEP 986 H= 986:00 +CPU= 29.481 20:42:26 STEP 987 H= 987:00 +CPU= 39.914 20:42:49 STEP 988 H= 988:00 +CPU= 21.616 20:43:15 STEP 989 H= 989:00 +CPU= 26.446 20:45:07 STEP 990 H= 990:00 +CPU= 46.261 20:45:38 STEP 991 H= 991:00 +CPU= 27.818 20:47:50 STEP 992 H= 992:00 +CPU= 31.892 20:52:02 STEP 993 H= 993:00 +CPU= 42.062 20:52:43 STEP 994 H= 994:00 +CPU= 32.260 20:53:14 STEP 995 H= 995:00 +CPU= 30.121 20:55:56 STEP 996 H= 996:00 +CPU= 52.237 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000996 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000996 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000996 20:57:17 STEP 997 H= 997:00 +CPU= 29.242 20:58:24 STEP 998 H= 998:00 +CPU= 28.064 20:59:14 STEP 999 H= 999:00 +CPU= 40.338 20:59:44 STEP 1000 H=1000:00 +CPU= 29.635 21:00:37 STEP 1001 H=1001:00 +CPU= 27.575 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0409 (argv4) batchid: 971 (argv5) wuid: 12188053 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c56e0fce334a4c64ccdcbb8de2318749 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip ic_ancil_file: ic_ancil_12188053 ifsdata_file: ifsdata_12188053 climate_data_file: clim_data_12188053 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_31ffd8be6e53a4d1f449eaffec38be52 to: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1223a2f6c8d1036a26a7bf58c824bcb6 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12188053.xml -- Model is restarting -- Adjusting last_iter, 1002, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml last_cpu_time: 42040 upload_file_number: 41 last_iter: 983 last_upload: 3542400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516 The child process has been launched with process id: 558281 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558281:558281] [20230107:210710:1673125630:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 21:09:08 STEP 984 H= 984:00 +CPU= 85.791 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0409 (argv4) batchid: 971 (argv5) wuid: 12188053 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c56e0fce334a4c64ccdcbb8de2318749 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip ic_ancil_file: ic_ancil_12188053 ifsdata_file: ifsdata_12188053 climate_data_file: clim_data_12188053 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_31ffd8be6e53a4d1f449eaffec38be52 to: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1223a2f6c8d1036a26a7bf58c824bcb6 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12188053.xml -- Model is restarting -- Adjusting last_iter, 985, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml last_cpu_time: 42165 upload_file_number: 41 last_iter: 983 last_upload: 3542400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516 The child process has been launched with process id: 558376 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558376:558376] [20230107:211853:1673126333:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0409 (argv4) batchid: 971 (argv5) wuid: 12188053 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c56e0fce334a4c64ccdcbb8de2318749 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip ic_ancil_file: ic_ancil_12188053 ifsdata_file: ifsdata_12188053 climate_data_file: clim_data_12188053 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_31ffd8be6e53a4d1f449eaffec38be52 to: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1223a2f6c8d1036a26a7bf58c824bcb6 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12188053.xml -- Model is restarting -- Adjusting last_iter, 984, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml last_cpu_time: 42232 upload_file_number: 41 last_iter: 983 last_upload: 3542400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516 The child process has been launched with process id: 558735 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:558735:558735] [20230107:231444:1673133284:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:42:22 STEP 984 H= 984:00 +CPU= 73.015 08:48:09 STEP 985 H= 985:00 +CPU= 43.467 08:48:48 STEP 986 H= 986:00 +CPU= 37.620 08:49:50 STEP 987 H= 987:00 +CPU= 60.324 08:50:39 STEP 988 H= 988:00 +CPU= 46.932 08:51:26 STEP 989 H= 989:00 +CPU= 45.189 08:52:34 STEP 990 H= 990:00 +CPU= 66.108 08:53:18 STEP 991 H= 991:00 +CPU= 43.347 08:54:04 STEP 992 H= 992:00 +CPU= 45.350 08:55:11 STEP 993 H= 993:00 +CPU= 66.157 08:55:59 STEP 994 H= 994:00 +CPU= 47.212 08:56:45 STEP 995 H= 995:00 +CPU= 44.997 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0409 (argv4) batchid: 971 (argv5) wuid: 12188053 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c56e0fce334a4c64ccdcbb8de2318749 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip ic_ancil_file: ic_ancil_12188053 ifsdata_file: ifsdata_12188053 climate_data_file: clim_data_12188053 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_31ffd8be6e53a4d1f449eaffec38be52 to: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1223a2f6c8d1036a26a7bf58c824bcb6 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12188053.xml -- Model is restarting -- Adjusting last_iter, 996, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml last_cpu_time: 42895 upload_file_number: 41 last_iter: 983 last_upload: 3542400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516 The child process has been launched with process id: 569834 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:569834:569834] [20230109:155001:1673279401:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:29:27 STEP 984 H= 984:00 +CPU= 73.280 08:43:48 STEP 985 H= 985:00 +CPU= 44.560 08:44:25 STEP 986 H= 986:00 +CPU= 35.945 08:45:22 STEP 987 H= 987:00 +CPU= 55.476 08:46:33 STEP 988 H= 988:00 +CPU= 39.109 08:47:20 STEP 989 H= 989:00 +CPU= 45.808 08:49:18 STEP 990 H= 990:00 +CPU= 75.021 08:50:08 STEP 991 H= 991:00 +CPU= 49.571 08:50:55 STEP 992 H= 992:00 +CPU= 46.582 08:52:55 STEP 993 H= 993:00 +CPU= 78.969 08:53:51 STEP 994 H= 994:00 +CPU= 54.837 09:00:43 STEP 995 H= 995:00 +CPU= 48.360 09:01:49 STEP 996 H= 996:00 +CPU= 45.185 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000996 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000996 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000996 09:02:13 STEP 997 H= 997:00 +CPU= 23.798 09:02:51 STEP 998 H= 998:00 +CPU= 37.759 09:03:50 STEP 999 H= 999:00 +CPU= 57.616 09:04:27 STEP 1000 H=1000:00 +CPU= 36.125 09:04:59 STEP 1001 H=1001:00 +CPU= 31.696 09:05:59 STEP 1002 H=1002:00 +CPU= 58.775 09:06:40 STEP 1003 H=1003:00 +CPU= 33.319 09:07:23 STEP 1004 H=1004:00 +CPU= 42.906 09:08:24 STEP 1005 H=1005:00 +CPU= 58.341 09:09:02 STEP 1006 H=1006:00 +CPU= 37.753 09:09:44 STEP 1007 H=1007:00 +CPU= 35.086 09:10:47 STEP 1008 H=1008:00 +CPU= 54.456 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001008 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001008 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMGGhq0f+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMSHhq0f+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053/ICMUAhq0f+000996 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516_41.zip Uploading the intermediate file: upload_file_41.zip Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0409 (argv4) batchid: 971 (argv5) wuid: 12188053 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188053 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c56e0fce334a4c64ccdcbb8de2318749 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0409_2002050100_123_971_12188053.zip ic_ancil_file: ic_ancil_12188053 ifsdata_file: ifsdata_12188053 climate_data_file: clim_data_12188053 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_31ffd8be6e53a4d1f449eaffec38be52 to: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12188053.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1223a2f6c8d1036a26a7bf58c824bcb6 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12188053.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12188053.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12188053.xml -- Model is restarting -- Adjusting last_iter, 1009, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12188053.xml last_cpu_time: 44102 upload_file_number: 42 last_iter: 1007 last_upload: 3628800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0409_2002050100_123_971_12188053_0_r848161516 The child process has been launched with process id: 578399 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Debian12:1:1:578399:578399] [20230110:091558:1673342158:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 09:17:26 STEP 1008 H=1008:00 +CPU= 69.610 09:18:06 STEP 1009 H=1009:00 +CPU= 38.842 09:18:44 STEP 1010 H=1010:00 +CPU= 37.840 09:19:38 STEP 1011 H=1011:00 +CPU= 53.315 09:20:15 STEP 1012 H=1012:00 +CPU= 36.434 09:20:49 STEP 1013 H=1013:00 +CPU= 33.255 09:21:46 STEP 1014 H=1014:00 +CPU= 56.732 09:22:23 STEP 1015 H=1015:00 +CPU= 36.833 09:23:01 STEP 1016 H=1016:00 +CPU= 37.621 09:24:04 STEP 1017 H=1017:00 +CPU= 61.889 09:24:46 STEP 1018 H=1018:00 +CPU= 41.074 09:25:26 STEP 1019 H=1019:00 +CPU= 39.217 09:26:35 STEP 1020 H=1020:00 +CPU= 68.010 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001020 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001020 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001020 09:27:09 STEP 1021 H=1021:00 +CPU= 34.452 09:27:45 STEP 1022 H=1022:00 +CPU= 35.043 09:28:39 STEP 1023 H=1023:00 +CPU= 53.028 09:29:14 STEP 1024 H=1024:00 +CPU= 34.648 09:29:46 STEP 1025 H=1025:00 +CPU= 31.785 09:30:38 STEP 1026 H=1026:00 +CPU= 52.203 09:31:15 STEP 1027 H=1027:00 +CPU= 36.189 09:31:53 STEP 1028 H=1028:00 +CPU= 36.892 09:32:54 STEP 1029 H=1029:00 +CPU= 60.735 09:33:38 STEP 1030 H=1030:00 +CPU= 43.584 09:34:23 STEP 1031 H=1031:00 +CPU= 44.185 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:11:35 | 1491147 | 22273520 | oifs_43r3_ps_0409_2002050100_123_971_12188053_0 | 864,000 | 9,718 | 0.0112 |
09 May 2023 16:11:35 | 1491147 | 22273520 | oifs_43r3_ps_0409_2002050100_123_971_12188053_0 | 864,000 | 9,718 | 0.0112 |
©2024 cpdn.org