<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
0 +CPU= 39.584
18:43:26 STEP 772 H= 772:00 +CPU= 22.719
18:43:48 STEP 773 H= 773:00 +CPU= 22.086
18:44:25 STEP 774 H= 774:00 +CPU= 37.451
18:44:48 STEP 775 H= 775:00 +CPU= 22.295
18:45:10 STEP 776 H= 776:00 +CPU= 22.803
18:45:50 STEP 777 H= 777:00 +CPU= 39.477
18:46:13 STEP 778 H= 778:00 +CPU= 22.823
18:46:37 STEP 779 H= 779:00 +CPU= 23.762
18:47:19 STEP 780 H= 780:00 +CPU= 42.436
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000780
18:47:41 STEP 781 H= 781:00 +CPU= 22.073
18:48:09 STEP 782 H= 782:00 +CPU= 27.341
18:48:44 STEP 783 H= 783:00 +CPU= 35.199
18:49:07 STEP 784 H= 784:00 +CPU= 22.804
18:49:35 STEP 785 H= 785:00 +CPU= 27.906
18:50:12 STEP 786 H= 786:00 +CPU= 36.636
18:50:34 STEP 787 H= 787:00 +CPU= 22.007
18:51:02 STEP 788 H= 788:00 +CPU= 28.116
18:51:40 STEP 789 H= 789:00 +CPU= 37.350
18:52:02 STEP 790 H= 790:00 +CPU= 22.462
18:52:29 STEP 791 H= 791:00 +CPU= 27.072
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_31.zip
Uploading the intermediate file: upload_file_31.zip
18:53:11 STEP 792 H= 792:00 +CPU= 41.925
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000792
18:53:33 STEP 793 H= 793:00 +CPU= 21.768
18:54:01 STEP 794 H= 794:00 +CPU= 27.846
18:54:35 STEP 795 H= 795:00 +CPU= 34.375
18:54:58 STEP 796 H= 796:00 +CPU= 22.986
18:55:24 STEP 797 H= 797:00 +CPU= 26.063
18:56:00 STEP 798 H= 798:00 +CPU= 35.002
18:56:24 STEP 799 H= 799:00 +CPU= 23.960
18:56:50 STEP 800 H= 800:00 +CPU= 26.646
18:57:24 STEP 801 H= 801:00 +CPU= 34.038
18:57:51 STEP 802 H= 802:00 +CPU= 26.280
18:58:14 STEP 803 H= 803:00 +CPU= 23.697
18:58:52 STEP 804 H= 804:00 +CPU= 37.758
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000804
18:59:20 STEP 805 H= 805:00 +CPU= 27.254
18:59:42 STEP 806 H= 806:00 +CPU= 22.913
19:00:16 STEP 807 H= 807:00 +CPU= 33.938
19:00:44 STEP 808 H= 808:00 +CPU= 27.663
19:01:06 STEP 809 H= 809:00 +CPU= 22.155
19:01:41 STEP 810 H= 810:00 +CPU= 34.847
19:02:07 STEP 811 H= 811:00 +CPU= 25.689
19:02:30 STEP 812 H= 812:00 +CPU= 22.836
19:03:05 STEP 813 H= 813:00 +CPU= 35.322
19:03:33 STEP 814 H= 814:00 +CPU= 27.326
19:03:56 STEP 815 H= 815:00 +CPU= 23.190
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_32.zip
Uploading the intermediate file: upload_file_32.zip
19:04:41 STEP 816 H= 816:00 +CPU= 45.225
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000816
19:05:04 STEP 817 H= 817:00 +CPU= 22.285
19:05:26 STEP 818 H= 818:00 +CPU= 22.784
19:06:06 STEP 819 H= 819:00 +CPU= 39.162
19:06:28 STEP 820 H= 820:00 +CPU= 22.577
19:06:51 STEP 821 H= 821:00 +CPU= 22.173
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0597
(argv4) batchid: 972
(argv5) wuid: 12189241
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fbfb89c0f1f86ecd944a7c817c4a887b to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0597_2003050100_123_972_12189241.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0597_2003050100_123_972_12189241.zip
ic_ancil_file: ic_ancil_12189241
ifsdata_file: ifsdata_12189241
climate_data_file: clim_data_12189241
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f3d56ae969701ee0fc86b4a5908afa8 to: /var/lib/boinc-client/slots/4/ic_ancil_12189241.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12189241.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a0c91ef6967e6fe8083cd6fee8fc3333 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12189241.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12189241.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12189241.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12189241.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12189241.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12189241.xml
-- Model is restarting --
Adjusting last_iter, 822, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12189241.xml
last_cpu_time: 24244
upload_file_number: 33
last_iter: 815
last_upload: 2934000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337
The child process has been launched with process id: 7193
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:7193:7193] [20230118:190829:1674097709:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
19:09:14 STEP 816 H= 816:00 +CPU= 41.647
19:09:36 STEP 817 H= 817:00 +CPU= 22.261
19:10:03 STEP 818 H= 818:00 +CPU= 26.731
19:10:38 STEP 819 H= 819:00 +CPU= 34.877
19:11:01 STEP 820 H= 820:00 +CPU= 23.094
19:11:27 STEP 821 H= 821:00 +CPU= 26.412
19:12:02 STEP 822 H= 822:00 +CPU= 34.992
19:12:25 STEP 823 H= 823:00 +CPU= 22.069
19:12:53 STEP 824 H= 824:00 +CPU= 28.666
19:13:27 STEP 825 H= 825:00 +CPU= 33.850
19:13:50 STEP 826 H= 826:00 +CPU= 22.996
19:14:16 STEP 827 H= 827:00 +CPU= 25.916
19:14:54 STEP 828 H= 828:00 +CPU= 37.782
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000828
19:15:19 STEP 829 H= 829:00 +CPU= 25.125
19:15:45 STEP 830 H= 830:00 +CPU= 25.521
19:16:19 STEP 831 H= 831:00 +CPU= 33.998
19:16:45 STEP 832 H= 832:00 +CPU= 25.650
19:17:10 STEP 833 H= 833:00 +CPU= 25.431
19:17:45 STEP 834 H= 834:00 +CPU= 35.105
19:18:09 STEP 835 H= 835:00 +CPU= 24.108
19:18:35 STEP 836 H= 836:00 +CPU= 25.694
19:19:09 STEP 837 H= 837:00 +CPU= 33.857
19:19:36 STEP 838 H= 838:00 +CPU= 26.546
19:19:59 STEP 839 H= 839:00 +CPU= 23.898
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_33.zip
Uploading the intermediate file: upload_file_33.zip
19:20:40 STEP 840 H= 840:00 +CPU= 40.244
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000840
19:21:07 STEP 841 H= 841:00 +CPU= 27.071
19:21:29 STEP 842 H= 842:00 +CPU= 22.405
19:22:04 STEP 843 H= 843:00 +CPU= 34.252
19:22:30 STEP 844 H= 844:00 +CPU= 26.370
19:22:52 STEP 845 H= 845:00 +CPU= 21.789
19:23:28 STEP 846 H= 846:00 +CPU= 36.008
19:23:52 STEP 847 H= 847:00 +CPU= 24.404
19:24:15 STEP 848 H= 848:00 +CPU= 22.417
19:24:50 STEP 849 H= 849:00 +CPU= 34.787
19:25:17 STEP 850 H= 850:00 +CPU= 27.004
19:25:39 STEP 851 H= 851:00 +CPU= 22.260
19:26:20 STEP 852 H= 852:00 +CPU= 40.907
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000852
19:26:43 STEP 853 H= 853:00 +CPU= 23.112
19:27:06 STEP 854 H= 854:00 +CPU= 22.570
19:27:45 STEP 855 H= 855:00 +CPU= 39.040
19:28:08 STEP 856 H= 856:00 +CPU= 22.493
19:28:30 STEP 857 H= 857:00 +CPU= 21.770
19:29:09 STEP 858 H= 858:00 +CPU= 39.679
19:29:31 STEP 859 H= 859:00 +CPU= 21.929
19:29:54 STEP 860 H= 860:00 +CPU= 22.571
19:30:33 STEP 861 H= 861:00 +CPU= 39.157
19:30:56 STEP 862 H= 862:00 +CPU= 23.032
19:31:19 STEP 863 H= 863:00 +CPU= 22.674
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_34.zip
Uploading the intermediate file: upload_file_34.zip
19:32:05 STEP 864 H= 864:00 +CPU= 45.378
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000864
19:32:27 STEP 865 H= 865:00 +CPU= 22.073
19:32:49 STEP 866 H= 866:00 +CPU= 22.671
19:33:28 STEP 867 H= 867:00 +CPU= 38.953
19:33:51 STEP 868 H= 868:00 +CPU= 22.711
19:34:14 STEP 869 H= 869:00 +CPU= 22.554
19:34:52 STEP 870 H= 870:00 +CPU= 38.371
19:35:14 STEP 871 H= 871:00 +CPU= 21.908
19:35:38 STEP 872 H= 872:00 +CPU= 24.105
19:36:16 STEP 873 H= 873:00 +CPU= 37.578
19:36:39 STEP 874 H= 874:00 +CPU= 22.528
19:37:04 STEP 875 H= 875:00 +CPU= 25.196
19:37:46 STEP 876 H= 876:00 +CPU= 42.067
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000876
19:38:08 STEP 877 H= 877:00 +CPU= 21.829
19:38:34 STEP 878 H= 878:00 +CPU= 25.945
19:39:10 STEP 879 H= 879:00 +CPU= 35.487
19:39:32 STEP 880 H= 880:00 +CPU= 22.428
19:40:00 STEP 881 H= 881:00 +CPU= 27.569
19:40:35 STEP 882 H= 882:00 +CPU= 35.191
19:40:57 STEP 883 H= 883:00 +CPU= 22.043
19:41:25 STEP 884 H= 884:00 +CPU= 27.803
19:41:59 STEP 885 H= 885:00 +CPU= 34.012
19:42:22 STEP 886 H= 886:00 +CPU= 22.507
19:42:49 STEP 887 H= 887:00 +CPU= 27.337
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_35.zip
Uploading the intermediate file: upload_file_35.zip
19:43:29 STEP 888 H= 888:00 +CPU= 40.387
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000888
19:43:53 STEP 889 H= 889:00 +CPU= 24.010
19:44:17 STEP 890 H= 890:00 +CPU= 23.960
19:44:51 STEP 891 H= 891:00 +CPU= 33.786
19:45:17 STEP 892 H= 892:00 +CPU= 25.404
19:45:41 STEP 893 H= 893:00 +CPU= 23.876
19:46:15 STEP 894 H= 894:00 +CPU= 34.448
19:46:42 STEP 895 H= 895:00 +CPU= 26.159
19:47:06 STEP 896 H= 896:00 +CPU= 23.950
19:47:40 STEP 897 H= 897:00 +CPU= 34.008
19:48:08 STEP 898 H= 898:00 +CPU= 28.321
19:48:30 STEP 899 H= 899:00 +CPU= 22.214
19:49:09 STEP 900 H= 900:00 +CPU= 38.300
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000900
19:49:36 STEP 901 H= 901:00 +CPU= 27.165
19:49:58 STEP 902 H= 902:00 +CPU= 22.634
19:50:34 STEP 903 H= 903:00 +CPU= 35.473
19:51:01 STEP 904 H= 904:00 +CPU= 26.748
19:51:24 STEP 905 H= 905:00 +CPU= 22.464
19:52:02 STEP 906 H= 906:00 +CPU= 38.549
19:52:27 STEP 907 H= 907:00 +CPU= 24.250
19:52:50 STEP 908 H= 908:00 +CPU= 22.776
19:53:29 STEP 909 H= 909:00 +CPU= 39.132
19:53:53 STEP 910 H= 910:00 +CPU= 23.162
19:54:15 STEP 911 H= 911:00 +CPU= 22.767
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_36.zip
Uploading the intermediate file: upload_file_36.zip
19:54:58 STEP 912 H= 912:00 +CPU= 42.244
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000912
19:55:20 STEP 913 H= 913:00 +CPU= 21.892
19:55:43 STEP 914 H= 914:00 +CPU= 23.040
19:56:22 STEP 915 H= 915:00 +CPU= 39.111
19:56:45 STEP 916 H= 916:00 +CPU= 22.576
19:57:08 STEP 917 H= 917:00 +CPU= 22.908
19:57:46 STEP 918 H= 918:00 +CPU= 38.579
19:58:08 STEP 919 H= 919:00 +CPU= 22.108
19:58:34 STEP 920 H= 920:00 +CPU= 25.315
19:59:10 STEP 921 H= 921:00 +CPU= 36.585
19:59:33 STEP 922 H= 922:00 +CPU= 22.448
19:59:58 STEP 923 H= 923:00 +CPU= 25.402
20:00:41 STEP 924 H= 924:00 +CPU= 42.232
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000924
20:01:03 STEP 925 H= 925:00 +CPU= 21.833
20:01:29 STEP 926 H= 926:00 +CPU= 26.465
20:02:05 STEP 927 H= 927:00 +CPU= 35.790
20:02:28 STEP 928 H= 928:00 +CPU= 22.585
20:02:54 STEP 929 H= 929:00 +CPU= 25.930
20:03:32 STEP 930 H= 930:00 +CPU= 38.081
20:03:54 STEP 931 H= 931:00 +CPU= 21.842
20:04:20 STEP 932 H= 932:00 +CPU= 25.656
20:04:55 STEP 933 H= 933:00 +CPU= 35.480
20:05:18 STEP 934 H= 934:00 +CPU= 22.366
20:05:46 STEP 935 H= 935:00 +CPU= 28.147
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_37.zip
Uploading the intermediate file: upload_file_37.zip
20:06:27 STEP 936 H= 936:00 +CPU= 40.784
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000936
20:06:49 STEP 937 H= 937:00 +CPU= 22.267
20:07:16 STEP 938 H= 938:00 +CPU= 27.111
20:07:50 STEP 939 H= 939:00 +CPU= 34.066
20:08:14 STEP 940 H= 940:00 +CPU= 23.228
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0597
(argv4) batchid: 972
(argv5) wuid: 12189241
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fbfb89c0f1f86ecd944a7c817c4a887b to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0597_2003050100_123_972_12189241.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0597_2003050100_123_972_12189241.zip
ic_ancil_file: ic_ancil_12189241
ifsdata_file: ifsdata_12189241
climate_data_file: clim_data_12189241
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f3d56ae969701ee0fc86b4a5908afa8 to: /var/lib/boinc-client/slots/4/ic_ancil_12189241.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12189241.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a0c91ef6967e6fe8083cd6fee8fc3333 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12189241.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12189241.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12189241.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12189241.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12189241.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12189241.xml
-- Model is restarting --
Adjusting last_iter, 941, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12189241.xml
last_cpu_time: 27825
upload_file_number: 38
last_iter: 935
last_upload: 3366000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337
The child process has been launched with process id: 8334
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:200934:1674101374:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
20:10:20 STEP 936 H= 936:00 +CPU= 42.198
20:10:43 STEP 937 H= 937:00 +CPU= 22.506
20:11:10 STEP 938 H= 938:00 +CPU= 27.370
20:11:45 STEP 939 H= 939:00 +CPU= 34.754
20:12:07 STEP 940 H= 940:00 +CPU= 22.522
20:12:35 STEP 941 H= 941:00 +CPU= 27.149
20:13:10 STEP 942 H= 942:00 +CPU= 34.971
20:13:32 STEP 943 H= 943:00 +CPU= 22.017
20:14:00 STEP 944 H= 944:00 +CPU= 28.072
20:14:34 STEP 945 H= 945:00 +CPU= 34.420
20:14:58 STEP 946 H= 946:00 +CPU= 23.221
20:15:24 STEP 947 H= 947:00 +CPU= 25.817
20:16:02 STEP 948 H= 948:00 +CPU= 38.208
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000948
20:16:27 STEP 949 H= 949:00 +CPU= 25.389
20:16:51 STEP 950 H= 950:00 +CPU= 23.806
20:17:26 STEP 951 H= 951:00 +CPU= 34.433
20:17:54 STEP 952 H= 952:00 +CPU= 28.288
20:18:18 STEP 953 H= 953:00 +CPU= 24.158
20:18:53 STEP 954 H= 954:00 +CPU= 34.914
20:19:17 STEP 955 H= 955:00 +CPU= 23.468
20:19:42 STEP 956 H= 956:00 +CPU= 24.936
20:20:16 STEP 957 H= 957:00 +CPU= 34.210
20:20:43 STEP 958 H= 958:00 +CPU= 26.947
20:21:09 STEP 959 H= 959:00 +CPU= 25.941
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_38.zip
Uploading the intermediate file: upload_file_38.zip
20:21:50 STEP 960 H= 960:00 +CPU= 40.414
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000960
20:22:16 STEP 961 H= 961:00 +CPU= 26.404
20:22:42 STEP 962 H= 962:00 +CPU= 25.349
20:23:16 STEP 963 H= 963:00 +CPU= 34.085
20:23:41 STEP 964 H= 964:00 +CPU= 25.388
20:24:04 STEP 965 H= 965:00 +CPU= 23.099
20:24:39 STEP 966 H= 966:00 +CPU= 35.010
20:25:06 STEP 967 H= 967:00 +CPU= 26.648
20:25:29 STEP 968 H= 968:00 +CPU= 23.034
20:26:04 STEP 969 H= 969:00 +CPU= 34.508
20:26:33 STEP 970 H= 970:00 +CPU= 29.502
20:26:57 STEP 971 H= 971:00 +CPU= 23.505
20:27:35 STEP 972 H= 972:00 +CPU= 38.020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000972
20:28:01 STEP 973 H= 973:00 +CPU= 25.853
20:28:24 STEP 974 H= 974:00 +CPU= 22.676
20:28:58 STEP 975 H= 975:00 +CPU= 34.101
20:29:25 STEP 976 H= 976:00 +CPU= 27.268
20:29:47 STEP 977 H= 977:00 +CPU= 22.011
20:30:22 STEP 978 H= 978:00 +CPU= 35.085
20:30:50 STEP 979 H= 979:00 +CPU= 27.010
20:31:12 STEP 980 H= 980:00 +CPU= 22.531
20:31:48 STEP 981 H= 981:00 +CPU= 35.595
20:32:14 STEP 982 H= 982:00 +CPU= 25.992
20:32:37 STEP 983 H= 983:00 +CPU= 22.937
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_39.zip
Uploading the intermediate file: upload_file_39.zip
20:33:23 STEP 984 H= 984:00 +CPU= 45.880
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000984
20:33:48 STEP 985 H= 985:00 +CPU= 25.419
20:34:11 STEP 986 H= 986:00 +CPU= 22.435
20:34:48 STEP 987 H= 987:00 +CPU= 37.390
20:35:13 STEP 988 H= 988:00 +CPU= 24.800
20:35:35 STEP 989 H= 989:00 +CPU= 21.762
20:36:14 STEP 990 H= 990:00 +CPU= 39.051
20:36:39 STEP 991 H= 991:00 +CPU= 25.282
20:37:02 STEP 992 H= 992:00 +CPU= 22.562
20:37:40 STEP 993 H= 993:00 +CPU= 37.792
20:38:05 STEP 994 H= 994:00 +CPU= 25.078
20:38:27 STEP 995 H= 995:00 +CPU= 22.130
20:39:10 STEP 996 H= 996:00 +CPU= 42.989
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000996
20:39:32 STEP 997 H= 997:00 +CPU= 21.869
20:39:55 STEP 998 H= 998:00 +CPU= 22.450
20:40:34 STEP 999 H= 999:00 +CPU= 39.457
20:40:57 STEP 1000 H=1000:00 +CPU= 22.939
20:41:19 STEP 1001 H=1001:00 +CPU= 21.754
20:41:59 STEP 1002 H=1002:00 +CPU= 39.676
20:42:21 STEP 1003 H=1003:00 +CPU= 21.990
20:42:43 STEP 1004 H=1004:00 +CPU= 22.416
20:43:23 STEP 1005 H=1005:00 +CPU= 39.439
20:43:45 STEP 1006 H=1006:00 +CPU= 22.646
20:44:08 STEP 1007 H=1007:00 +CPU= 22.668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_40.zip
Uploading the intermediate file: upload_file_40.zip
20:44:54 STEP 1008 H=1008:00 +CPU= 45.505
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001008
20:45:15 STEP 1009 H=1009:00 +CPU= 21.689
20:45:40 STEP 1010 H=1010:00 +CPU= 24.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:46:59 STEP 1011 H=1011:00 +CPU= 44.196
20:47:31 STEP 1012 H=1012:00 +CPU= 29.054
20:47:53 STEP 1013 H=1013:00 +CPU= 22.503
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4035MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.738] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4035 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.739] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADINTG
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.743] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADINTG-RADLSW (1210)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.743] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADIATION_SCHEME
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.743] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_interface:radiation
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204807:1674103687:2312.743] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_mcica_lw:solver_mcica_lw
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204808:1674103688:2313.743] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8334:8334] [20230118:204808:1674103688:2313.743] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 000000000188BC0B radiation_two_str 284 radiation_two_stream.F90
oifs_43r3_model.e 000000000197279F radiation_mcica_l 159 radiation_mcica_lw.F90
oifs_43r3_model.e 0000000001862B6D radiation_interfa 387 radiation_interface.F90
oifs_43r3_model.e 00000000012D30E5 radiation_scheme_ 484 radiation_scheme.F90
oifs_43r3_model.e 000000000129A415 radintg_ 1455 radintg.F90
oifs_43r3_model.e 0000000001270124 raddrv_ 537 raddrv.F90
oifs_43r3_model.e 0000000000BFD839 gp_model_ 482 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0597
(argv4) batchid: 972
(argv5) wuid: 12189241
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fbfb89c0f1f86ecd944a7c817c4a887b to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0597_2003050100_123_972_12189241.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0597_2003050100_123_972_12189241.zip
ic_ancil_file: ic_ancil_12189241
ifsdata_file: ifsdata_12189241
climate_data_file: clim_data_12189241
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4f3d56ae969701ee0fc86b4a5908afa8 to: /var/lib/boinc-client/slots/4/ic_ancil_12189241.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12189241.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a0c91ef6967e6fe8083cd6fee8fc3333 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12189241.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12189241.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12189241.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12189241.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12189241.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12189241.xml
-- Model is restarting --
Adjusting last_iter, 1014, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12189241.xml
last_cpu_time: 30089
upload_file_number: 41
last_iter: 1007
last_upload: 3625200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337
The child process has been launched with process id: 2627
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:2627:2627] [20230120:155831:1674259111:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:04:49 STEP 1008 H=1008:00 +CPU= 66.149
16:05:11 STEP 1009 H=1009:00 +CPU= 22.401
16:05:35 STEP 1010 H=1010:00 +CPU= 23.129
16:06:16 STEP 1011 H=1011:00 +CPU= 39.981
16:06:39 STEP 1012 H=1012:00 +CPU= 23.211
16:07:05 STEP 1013 H=1013:00 +CPU= 24.432
16:07:47 STEP 1014 H=1014:00 +CPU= 40.826
16:08:12 STEP 1015 H=1015:00 +CPU= 24.439
16:08:41 STEP 1016 H=1016:00 +CPU= 28.676
16:09:19 STEP 1017 H=1017:00 +CPU= 37.535
16:09:43 STEP 1018 H=1018:00 +CPU= 23.253
16:10:10 STEP 1019 H=1019:00 +CPU= 27.227
16:10:53 STEP 1020 H=1020:00 +CPU= 41.764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001020
16:11:15 STEP 1021 H=1021:00 +CPU= 22.157
16:11:43 STEP 1022 H=1022:00 +CPU= 27.788
16:12:21 STEP 1023 H=1023:00 +CPU= 37.626
16:12:46 STEP 1024 H=1024:00 +CPU= 23.998
16:13:15 STEP 1025 H=1025:00 +CPU= 28.296
16:13:53 STEP 1026 H=1026:00 +CPU= 37.418
16:14:15 STEP 1027 H=1027:00 +CPU= 22.079
16:14:43 STEP 1028 H=1028:00 +CPU= 27.653
16:15:20 STEP 1029 H=1029:00 +CPU= 36.954
16:15:43 STEP 1030 H=1030:00 +CPU= 22.693
16:16:09 STEP 1031 H=1031:00 +CPU= 26.155
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189241/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0597_2003050100_123_972_12189241_0_r444748337_41.zip
Uploading the intermediate file: upload_file_41.zip
16:16:55 STEP 1032 H=1032:00 +CPU= 45.118
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001032
16:17:18 STEP 1033 H=1033:00 +CPU= 22.941
16:17:46 STEP 1034 H=1034:00 +CPU= 27.934
16:18:22 STEP 1035 H=1035:00 +CPU= 35.899
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
