Name | oifs_43r3_ps_0562_1988050100_123_957_12174206_0 |
Workunit | 12174206 |
Created | 21 Dec 2022, 14:14:06 UTC |
Sent | 21 Dec 2022, 14:26:19 UTC |
Report deadline | 20 Jan 2023, 14:26:19 UTC |
Received | 22 Dec 2022, 13:02:33 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1509499 |
Run time | 6 hours 46 min 46 sec |
CPU time | 6 hours 39 min 28 sec |
Validate state | Invalid |
Credit | 2,419.20 |
Device peak FLOPS | 5.06 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,416.81 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 7,681.86 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> /home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:35929:35929] [20221221:213717:1671655037:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:35929:35929] [20221221:213717:1671655037:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:35929:35929] [20221221:213717:1671655037:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:35929:35929] [20221221:213717:1671655037:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 21:37:50 STEP 816 H= 816:00 +CPU= 31.436 21:38:08 STEP 817 H= 817:00 +CPU= 17.731 21:38:27 STEP 818 H= 818:00 +CPU= 18.240 21:38:55 STEP 819 H= 819:00 +CPU= 28.089 21:39:13 STEP 820 H= 820:00 +CPU= 18.272 21:39:31 STEP 821 H= 821:00 +CPU= 17.721 21:40:00 STEP 822 H= 822:00 +CPU= 28.763 21:40:18 STEP 823 H= 823:00 +CPU= 17.810 21:40:36 STEP 824 H= 824:00 +CPU= 18.386 21:41:05 STEP 825 H= 825:00 +CPU= 28.195 21:41:23 STEP 826 H= 826:00 +CPU= 18.479 21:41:41 STEP 827 H= 827:00 +CPU= 17.885 21:42:13 STEP 828 H= 828:00 +CPU= 31.104 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000828 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000828 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000828 21:42:31 STEP 829 H= 829:00 +CPU= 17.844 21:42:49 STEP 830 H= 830:00 +CPU= 18.382 21:43:17 STEP 831 H= 831:00 +CPU= 28.175 21:43:36 STEP 832 H= 832:00 +CPU= 18.339 21:43:54 STEP 833 H= 833:00 +CPU= 17.807 21:44:23 STEP 834 H= 834:00 +CPU= 28.786 21:44:41 STEP 835 H= 835:00 +CPU= 17.912 21:44:59 STEP 836 H= 836:00 +CPU= 18.480 21:45:28 STEP 837 H= 837:00 +CPU= 28.224 21:45:46 STEP 838 H= 838:00 +CPU= 18.454 21:46:05 STEP 839 H= 839:00 +CPU= 18.413 21:46:38 STEP 840 H= 840:00 +CPU= 32.794 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000840 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000840 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000828 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000828 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000828 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843_34.zip Uploading the intermediate file: upload_file_34.zip 21:46:56 STEP 841 H= 841:00 +CPU= 17.871 21:47:14 STEP 842 H= 842:00 +CPU= 18.374 21:47:43 STEP 843 H= 843:00 +CPU= 28.098 21:48:01 STEP 844 H= 844:00 +CPU= 18.479 21:48:19 STEP 845 H= 845:00 +CPU= 17.840 21:48:48 STEP 846 H= 846:00 +CPU= 28.689 21:49:06 STEP 847 H= 847:00 +CPU= 17.906 21:49:25 STEP 848 H= 848:00 +CPU= 18.427 21:49:53 STEP 849 H= 849:00 +CPU= 28.235 21:50:11 STEP 850 H= 850:00 +CPU= 18.451 21:50:29 STEP 851 H= 851:00 +CPU= 17.901 21:51:01 STEP 852 H= 852:00 +CPU= 31.113 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000852 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000852 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000852 21:51:19 STEP 853 H= 853:00 +CPU= 17.883 21:51:37 STEP 854 H= 854:00 +CPU= 18.426 21:52:06 STEP 855 H= 855:00 +CPU= 28.144 21:52:24 STEP 856 H= 856:00 +CPU= 18.418 21:52:42 STEP 857 H= 857:00 +CPU= 17.875 21:53:11 STEP 858 H= 858:00 +CPU= 28.674 21:53:29 STEP 859 H= 859:00 +CPU= 17.788 21:53:47 STEP 860 H= 860:00 +CPU= 18.303 21:54:16 STEP 861 H= 861:00 +CPU= 28.129 21:54:34 STEP 862 H= 862:00 +CPU= 18.273 21:54:52 STEP 863 H= 863:00 +CPU= 18.289 21:55:26 STEP 864 H= 864:00 +CPU= 32.924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000864 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000864 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000852 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000852 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000852 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843_35.zip Uploading the intermediate file: upload_file_35.zip 21:55:44 STEP 865 H= 865:00 +CPU= 17.801 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0562 (argv4) batchid: 957 (argv5) wuid: 12174206 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_956272283742ea0c4edf6e9fbe887e86 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip ic_ancil_file: ic_ancil_12174206 ifsdata_file: ifsdata_12174206 climate_data_file: clim_data_12174206 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_9aef573571f3c7074d4c78fb20b24e50 to: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4654495509b41bf47141d23e114291ed to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174206.xml -- Model is restarting -- Adjusting last_iter, 865, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml last_cpu_time: 20667 upload_file_number: 36 last_iter: 863 last_upload: 3110400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843 The child process has been launched with process id: 36102 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:36102:36102] [20221221:215636:1671656196:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 21:57:10 STEP 864 H= 864:00 +CPU= 31.792 21:57:28 STEP 865 H= 865:00 +CPU= 17.577 21:57:46 STEP 866 H= 866:00 +CPU= 18.290 21:58:15 STEP 867 H= 867:00 +CPU= 28.401 21:58:33 STEP 868 H= 868:00 +CPU= 18.179 21:58:51 STEP 869 H= 869:00 +CPU= 17.793 21:59:20 STEP 870 H= 870:00 +CPU= 28.988 21:59:38 STEP 871 H= 871:00 +CPU= 17.549 21:59:56 STEP 872 H= 872:00 +CPU= 18.341 22:00:25 STEP 873 H= 873:00 +CPU= 28.427 22:00:43 STEP 874 H= 874:00 +CPU= 18.151 22:01:01 STEP 875 H= 875:00 +CPU= 17.819 22:01:33 STEP 876 H= 876:00 +CPU= 31.316 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000876 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000876 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000876 22:01:50 STEP 877 H= 877:00 +CPU= 17.618 22:02:09 STEP 878 H= 878:00 +CPU= 18.403 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0562 (argv4) batchid: 957 (argv5) wuid: 12174206 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_956272283742ea0c4edf6e9fbe887e86 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip ic_ancil_file: ic_ancil_12174206 ifsdata_file: ifsdata_12174206 climate_data_file: clim_data_12174206 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_9aef573571f3c7074d4c78fb20b24e50 to: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4654495509b41bf47141d23e114291ed to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174206.xml -- Model is restarting -- Adjusting last_iter, 879, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml last_cpu_time: 21004 upload_file_number: 36 last_iter: 863 last_upload: 3110400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843 The child process has been launched with process id: 37216 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:220330:1671656610:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 22:04:03 STEP 864 H= 864:00 +CPU= 31.572 22:04:21 STEP 865 H= 865:00 +CPU= 17.865 22:04:40 STEP 866 H= 866:00 +CPU= 18.373 22:05:08 STEP 867 H= 867:00 +CPU= 28.230 22:05:27 STEP 868 H= 868:00 +CPU= 18.386 22:05:45 STEP 869 H= 869:00 +CPU= 17.881 22:06:14 STEP 870 H= 870:00 +CPU= 28.909 22:06:32 STEP 871 H= 871:00 +CPU= 17.919 22:06:50 STEP 872 H= 872:00 +CPU= 18.479 22:07:19 STEP 873 H= 873:00 +CPU= 28.379 22:07:38 STEP 874 H= 874:00 +CPU= 18.486 22:07:56 STEP 875 H= 875:00 +CPU= 17.943 22:08:27 STEP 876 H= 876:00 +CPU= 31.209 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000876 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000876 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000876 22:08:45 STEP 877 H= 877:00 +CPU= 17.949 22:09:04 STEP 878 H= 878:00 +CPU= 18.511 22:09:32 STEP 879 H= 879:00 +CPU= 28.270 22:09:51 STEP 880 H= 880:00 +CPU= 18.458 22:10:09 STEP 881 H= 881:00 +CPU= 17.953 22:10:38 STEP 882 H= 882:00 +CPU= 28.911 22:10:56 STEP 883 H= 883:00 +CPU= 17.922 22:11:14 STEP 884 H= 884:00 +CPU= 18.472 22:11:43 STEP 885 H= 885:00 +CPU= 28.217 22:12:01 STEP 886 H= 886:00 +CPU= 18.413 22:12:20 STEP 887 H= 887:00 +CPU= 18.341 22:12:53 STEP 888 H= 888:00 +CPU= 32.908 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000888 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000888 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000876 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843_36.zip Uploading the intermediate file: upload_file_36.zip 22:13:11 STEP 889 H= 889:00 +CPU= 17.867 22:13:29 STEP 890 H= 890:00 +CPU= 18.353 22:13:58 STEP 891 H= 891:00 +CPU= 28.348 22:14:16 STEP 892 H= 892:00 +CPU= 18.390 22:14:34 STEP 893 H= 893:00 +CPU= 17.887 22:15:04 STEP 894 H= 894:00 +CPU= 28.990 22:15:22 STEP 895 H= 895:00 +CPU= 17.904 22:15:40 STEP 896 H= 896:00 +CPU= 18.553 22:16:09 STEP 897 H= 897:00 +CPU= 28.276 22:16:27 STEP 898 H= 898:00 +CPU= 18.571 22:16:45 STEP 899 H= 899:00 +CPU= 17.928 22:17:17 STEP 900 H= 900:00 +CPU= 31.228 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000900 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000900 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000900 22:17:34 STEP 901 H= 901:00 +CPU= 17.375 22:17:52 STEP 902 H= 902:00 +CPU= 17.482 22:18:19 STEP 903 H= 903:00 +CPU= 27.056 22:18:37 STEP 904 H= 904:00 +CPU= 17.873 22:18:55 STEP 905 H= 905:00 +CPU= 17.632 22:19:24 STEP 906 H= 906:00 +CPU= 28.971 22:19:42 STEP 907 H= 907:00 +CPU= 17.806 22:20:00 STEP 908 H= 908:00 +CPU= 18.099 22:20:29 STEP 909 H= 909:00 +CPU= 28.321 22:20:47 STEP 910 H= 910:00 +CPU= 18.299 22:21:05 STEP 911 H= 911:00 +CPU= 18.070 22:21:38 STEP 912 H= 912:00 +CPU= 32.953 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000912 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000912 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000900 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843_37.zip Uploading the intermediate file: upload_file_37.zip 22:21:56 STEP 913 H= 913:00 +CPU= 17.798 22:22:15 STEP 914 H= 914:00 +CPU= 18.168 22:22:43 STEP 915 H= 915:00 +CPU= 28.276 22:23:02 STEP 916 H= 916:00 +CPU= 18.353 22:23:19 STEP 917 H= 917:00 +CPU= 17.601 22:23:48 STEP 918 H= 918:00 +CPU= 28.910 22:24:06 STEP 919 H= 919:00 +CPU= 17.817 22:24:25 STEP 920 H= 920:00 +CPU= 18.142 22:24:53 STEP 921 H= 921:00 +CPU= 28.276 22:25:11 STEP 922 H= 922:00 +CPU= 18.356 22:25:29 STEP 923 H= 923:00 +CPU= 17.612 22:26:00 STEP 924 H= 924:00 +CPU= 31.182 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000924 22:26:18 STEP 925 H= 925:00 +CPU= 17.838 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4749MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4749 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALL_SL [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 2 (1005) [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LAPINEB [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LARCINB [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LAITRE_GMV [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222623:1671657983:1373.785] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LAITRI [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222624:1671657984:1374.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:obata-H87-HD3:1:1:37216:37216] [20221221:222624:1671657984:1374.785] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 0000000000DD2368 laitri_ 352 laitri.F90 oifs_43r3_model.e 0000000000DDBEE4 laitre_gmv_ 143 laitre_gmv.F90 oifs_43r3_model.e 0000000000DBD8F4 larcinb_ 301 larcinb.F90 oifs_43r3_model.e 0000000000D7EBA8 lapineb_ 451 lapineb.F90 oifs_43r3_model.e 0000000000D7B935 call_sl_ 363 call_sl.F90 oifs_43r3_model.e 0000000000D784FD call_sl_heap_ 94 call_sl_heap.F90 oifs_43r3_model.e 0000000000BF9958 gp_model_ 533 gp_model.F90 oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90 oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90 oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0562 (argv4) batchid: 957 (argv5) wuid: 12174206 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_956272283742ea0c4edf6e9fbe887e86 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip ic_ancil_file: ic_ancil_12174206 ifsdata_file: ifsdata_12174206 climate_data_file: clim_data_12174206 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_9aef573571f3c7074d4c78fb20b24e50 to: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4654495509b41bf47141d23e114291ed to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174206.xml -- Model is restarting -- Adjusting last_iter, 926, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml last_cpu_time: 22367 upload_file_number: 38 last_iter: 911 last_upload: 3283200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843 The child process has been launched with process id: 2778 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:2778:2778] [20221222:114531:1671705931:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:46:19 STEP 912 H= 912:00 +CPU= 32.497 11:46:39 STEP 913 H= 913:00 +CPU= 17.968 11:46:58 STEP 914 H= 914:00 +CPU= 18.640 11:47:29 STEP 915 H= 915:00 +CPU= 29.581 11:47:50 STEP 916 H= 916:00 +CPU= 20.520 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:48:22 STEP 917 H= 917:00 +CPU= 19.092 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:57:57 STEP 918 H= 918:00 +CPU= 31.478 11:58:19 STEP 919 H= 919:00 +CPU= 19.324 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:58:51 STEP 920 H= 920:00 +CPU= 19.851 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:59:35 STEP 921 H= 921:00 +CPU= 30.031 11:59:55 STEP 922 H= 922:00 +CPU= 19.541 12:00:14 STEP 923 H= 923:00 +CPU= 18.706 12:00:49 STEP 924 H= 924:00 +CPU= 32.719 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000924 12:01:09 STEP 925 H= 925:00 +CPU= 18.913 12:01:30 STEP 926 H= 926:00 +CPU= 19.473 12:02:00 STEP 927 H= 927:00 +CPU= 29.017 12:02:19 STEP 928 H= 928:00 +CPU= 18.918 12:02:37 STEP 929 H= 929:00 +CPU= 18.199 12:03:07 STEP 930 H= 930:00 +CPU= 29.594 12:03:26 STEP 931 H= 931:00 +CPU= 18.358 12:03:45 STEP 932 H= 932:00 +CPU= 18.433 12:04:13 STEP 933 H= 933:00 +CPU= 28.338 12:04:32 STEP 934 H= 934:00 +CPU= 18.455 12:04:51 STEP 935 H= 935:00 +CPU= 18.517 12:05:26 STEP 936 H= 936:00 +CPU= 33.113 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMGGhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMSHhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206/ICMUAhq0f+000924 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843_38.zip Uploading the intermediate file: upload_file_38.zip 12:05:44 STEP 937 H= 937:00 +CPU= 18.134 12:06:03 STEP 938 H= 938:00 +CPU= 18.582 12:06:31 STEP 939 H= 939:00 +CPU= 28.379 12:06:50 STEP 940 H= 940:00 +CPU= 18.506 12:07:08 STEP 941 H= 941:00 +CPU= 17.913 12:07:37 STEP 942 H= 942:00 +CPU= 29.044 12:07:55 STEP 943 H= 943:00 +CPU= 18.011 12:08:14 STEP 944 H= 944:00 +CPU= 18.569 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:09:17 STEP 945 H= 945:00 +CPU= 30.040 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:05:48 STEP 946 H= 946:00 +CPU= 19.416 13:06:07 STEP 947 H= 947:00 +CPU= 18.248 13:06:39 STEP 948 H= 948:00 +CPU= 31.817 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000948 13:06:57 STEP 949 H= 949:00 +CPU= 18.248 13:07:17 STEP 950 H= 950:00 +CPU= 18.805 13:07:46 STEP 951 H= 951:00 +CPU= 28.818 13:08:05 STEP 952 H= 952:00 +CPU= 18.842 13:08:23 STEP 953 H= 953:00 +CPU= 18.191 13:08:53 STEP 954 H= 954:00 +CPU= 29.482 13:09:12 STEP 955 H= 955:00 +CPU= 18.289 13:09:31 STEP 956 H= 956:00 +CPU= 18.730 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0562 (argv4) batchid: 957 (argv5) wuid: 12174206 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174206 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_956272283742ea0c4edf6e9fbe887e86 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0562_1988050100_123_957_12174206.zip ic_ancil_file: ic_ancil_12174206 ifsdata_file: ifsdata_12174206 climate_data_file: clim_data_12174206 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_9aef573571f3c7074d4c78fb20b24e50 to: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12174206.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4654495509b41bf47141d23e114291ed to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12174206.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12174206.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12174206.xml -- Model is restarting -- Adjusting last_iter, 957, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12174206.xml last_cpu_time: 23408 upload_file_number: 39 last_iter: 935 last_upload: 3369600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0562_1988050100_123_957_12174206_0_r1442304843 The child process has been launched with process id: 10436 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:10436:10436] [20221222:131103:1671711063:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 13:11:45 STEP 936 H= 936:00 +CPU= 32.622 13:12:03 STEP 937 H= 937:00 +CPU= 17.849 13:12:21 STEP 938 H= 938:00 +CPU= 18.537 13:12:51 STEP 939 H= 939:00 +CPU= 28.894 13:13:09 STEP 940 H= 940:00 +CPU= 18.367 13:13:28 STEP 941 H= 941:00 +CPU= 18.099 13:13:58 STEP 942 H= 942:00 +CPU= 29.537 13:14:16 STEP 943 H= 943:00 +CPU= 17.864 13:14:35 STEP 944 H= 944:00 +CPU= 18.659 13:15:04 STEP 945 H= 945:00 +CPU= 28.984 13:15:23 STEP 946 H= 946:00 +CPU= 18.415 13:15:41 STEP 947 H= 947:00 +CPU= 18.126 13:16:14 STEP 948 H= 948:00 +CPU= 31.869 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000948 13:16:32 STEP 949 H= 949:00 +CPU= 17.976 13:16:51 STEP 950 H= 950:00 +CPU= 18.758 13:17:20 STEP 951 H= 951:00 +CPU= 28.936 13:17:39 STEP 952 H= 952:00 +CPU= 18.442 13:17:58 STEP 953 H= 953:00 +CPU= 18.193 13:18:27 STEP 954 H= 954:00 +CPU= 29.483 13:18:46 STEP 955 H= 955:00 +CPU= 17.923 13:19:05 STEP 956 H= 956:00 +CPU= 18.707 13:19:34 STEP 957 H= 957:00 +CPU= 28.915 13:19:53 STEP 958 H= 958:00 +CPU= 18.415 13:20:12 STEP 959 H= 959:00 +CPU= 18.700 Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 15:46:22 | 1509499 | 22258820 | oifs_43r3_ps_0562_1988050100_123_957_12174206_0 | 1,728,000 | 11,255 | 0.0065 |
©2024 climateprediction.net