climateprediction.net home page
Task 22290243

Task 22290243

Name oifs_43r3_ps_0129_2018050100_123_987_12203773_0
Workunit 12203773
Created 24 Dec 2022, 19:50:18 UTC
Sent 20 Jan 2023, 11:10:27 UTC
Report deadline 21 Mar 2023, 11:10:27 UTC
Received 24 Jan 2023, 22:57:04 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1511650
Run time 2 days 17 hours 20 min 49 sec
CPU time 2 days 6 hours 28 min 50 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 2.42 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.21 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,449.54 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
68.592
  23:28:47 STEP 1790 H=1790:00 +CPU= 71.341
  23:30:59 STEP 1791 H=1791:00 +CPU=111.770
  23:32:20 STEP 1792 H=1792:00 +CPU= 71.601
  23:33:40 STEP 1793 H=1793:00 +CPU= 68.809
  23:35:54 STEP 1794 H=1794:00 +CPU=115.048
  23:37:14 STEP 1795 H=1795:00 +CPU= 68.609
  23:38:37 STEP 1796 H=1796:00 +CPU= 71.495
  23:40:46 STEP 1797 H=1797:00 +CPU=111.997
  23:42:08 STEP 1798 H=1798:00 +CPU= 71.206
  23:43:32 STEP 1799 H=1799:00 +CPU= 70.917
  23:46:02 STEP 1800 H=1800:00 +CPU=130.174
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_74.zip
Uploading the intermediate file: upload_file_74.zip
  23:47:19 STEP 1801 H=1801:00 +CPU= 67.827
  23:48:41 STEP 1802 H=1802:00 +CPU= 71.062
  23:50:50 STEP 1803 H=1803:00 +CPU=111.843
  23:52:10 STEP 1804 H=1804:00 +CPU= 71.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:24 STEP 1805 H=1805:00 +CPU= 68.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:57:04 STEP 1806 H=1806:00 +CPU=115.285
  23:58:30 STEP 1807 H=1807:00 +CPU= 68.750
  23:59:55 STEP 1808 H=1808:00 +CPU= 71.387
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:05:18 STEP 1809 H=1809:00 +CPU=111.954
  00:06:42 STEP 1810 H=1810:00 +CPU= 71.264
  00:08:04 STEP 1811 H=1811:00 +CPU= 68.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:12:31 STEP 1812 H=1812:00 +CPU=125.239
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001812
  00:13:54 STEP 1813 H=1813:00 +CPU= 68.738
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:15:31 STEP 1814 H=1814:00 +CPU= 71.623
  00:17:41 STEP 1815 H=1815:00 +CPU=111.581
  00:19:01 STEP 1816 H=1816:00 +CPU= 71.368
  00:20:21 STEP 1817 H=1817:00 +CPU= 68.688
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:23:25 STEP 1818 H=1818:00 +CPU=114.663
  00:24:44 STEP 1819 H=1819:00 +CPU= 68.450
  00:26:11 STEP 1820 H=1820:00 +CPU= 71.271
  00:28:17 STEP 1821 H=1821:00 +CPU=110.975
  00:29:39 STEP 1822 H=1822:00 +CPU= 71.262
  00:31:01 STEP 1823 H=1823:00 +CPU= 71.281
  00:33:32 STEP 1824 H=1824:00 +CPU=130.831
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_75.zip
Uploading the intermediate file: upload_file_75.zip
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0129
(argv4) batchid: 987
(argv5) wuid: 12203773
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1e8fb4c21438175f276600405e822e57 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
ic_ancil_file: ic_ancil_12203773
ifsdata_file: ifsdata_12203773
climate_data_file: clim_data_12203773
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b616257e41b7fedeba0331227266438f to: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_203885341e62401100ca0e9a22aa1430 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203773.xml
-- Model is restarting --
Adjusting last_iter, 1825, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
last_cpu_time: 175445
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093
The child process has been launched with process id: 2957572
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2957572:2957572] [20230124:014250:1674524570:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:45:24 STEP 1824 H=1824:00 +CPU=127.206
  01:46:42 STEP 1825 H=1825:00 +CPU= 68.769
  01:48:06 STEP 1826 H=1826:00 +CPU= 71.612
  01:50:19 STEP 1827 H=1827:00 +CPU=111.155
  01:51:41 STEP 1828 H=1828:00 +CPU= 71.042
  01:53:00 STEP 1829 H=1829:00 +CPU= 68.234
  01:55:16 STEP 1830 H=1830:00 +CPU=113.982
  01:56:34 STEP 1831 H=1831:00 +CPU= 68.512
  01:57:56 STEP 1832 H=1832:00 +CPU= 71.104
  02:00:05 STEP 1833 H=1833:00 +CPU=111.434
  02:01:23 STEP 1834 H=1834:00 +CPU= 70.637
  02:02:42 STEP 1835 H=1835:00 +CPU= 68.400
  02:05:06 STEP 1836 H=1836:00 +CPU=124.224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001836
  02:06:26 STEP 1837 H=1837:00 +CPU= 68.329
  02:07:48 STEP 1838 H=1838:00 +CPU= 71.115
  02:09:54 STEP 1839 H=1839:00 +CPU=110.855
  02:11:14 STEP 1840 H=1840:00 +CPU= 70.901
  02:12:32 STEP 1841 H=1841:00 +CPU= 68.469
  02:14:44 STEP 1842 H=1842:00 +CPU=114.312
  02:16:02 STEP 1843 H=1843:00 +CPU= 68.513
  02:17:27 STEP 1844 H=1844:00 +CPU= 71.128
  02:19:36 STEP 1845 H=1845:00 +CPU=111.511
  02:20:59 STEP 1846 H=1846:00 +CPU= 71.230
  02:22:20 STEP 1847 H=1847:00 +CPU= 71.041
  02:24:50 STEP 1848 H=1848:00 +CPU=129.982
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_76.zip
Uploading the intermediate file: upload_file_76.zip
  02:26:10 STEP 1849 H=1849:00 +CPU= 68.501
  02:27:33 STEP 1850 H=1850:00 +CPU= 71.048
  02:29:44 STEP 1851 H=1851:00 +CPU=111.085
  02:31:06 STEP 1852 H=1852:00 +CPU= 71.324
  02:32:25 STEP 1853 H=1853:00 +CPU= 68.465
  02:34:40 STEP 1854 H=1854:00 +CPU=114.035
  02:35:56 STEP 1855 H=1855:00 +CPU= 67.865
  02:37:17 STEP 1856 H=1856:00 +CPU= 71.076
  02:39:27 STEP 1857 H=1857:00 +CPU=110.987
  02:40:49 STEP 1858 H=1858:00 +CPU= 71.299
  02:42:10 STEP 1859 H=1859:00 +CPU= 68.568
  02:44:35 STEP 1860 H=1860:00 +CPU=123.926
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001860
  02:45:53 STEP 1861 H=1861:00 +CPU= 67.812
  02:47:14 STEP 1862 H=1862:00 +CPU= 70.597
  02:49:24 STEP 1863 H=1863:00 +CPU=111.094
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0129
(argv4) batchid: 987
(argv5) wuid: 12203773
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1e8fb4c21438175f276600405e822e57 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
ic_ancil_file: ic_ancil_12203773
ifsdata_file: ifsdata_12203773
climate_data_file: clim_data_12203773
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b616257e41b7fedeba0331227266438f to: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_203885341e62401100ca0e9a22aa1430 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203773.xml
-- Model is restarting --
Adjusting last_iter, 1864, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
last_cpu_time: 178943
upload_file_number: 77
last_iter: 1847
last_upload: 6652800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093
The child process has been launched with process id: 2961295
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:2961295:2961295] [20230124:025904:1674529144:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:01:36 STEP 1848 H=1848:00 +CPU=126.649
  03:02:56 STEP 1849 H=1849:00 +CPU= 68.350
  03:04:18 STEP 1850 H=1850:00 +CPU= 70.900
  03:06:26 STEP 1851 H=1851:00 +CPU=110.948
  03:07:50 STEP 1852 H=1852:00 +CPU= 70.887
  03:09:12 STEP 1853 H=1853:00 +CPU= 67.910
  03:11:26 STEP 1854 H=1854:00 +CPU=114.273
  03:12:46 STEP 1855 H=1855:00 +CPU= 68.431
  03:14:08 STEP 1856 H=1856:00 +CPU= 71.020
  03:16:14 STEP 1857 H=1857:00 +CPU=110.829
  03:17:35 STEP 1858 H=1858:00 +CPU= 70.989
  03:18:50 STEP 1859 H=1859:00 +CPU= 67.871
  03:21:14 STEP 1860 H=1860:00 +CPU=123.911
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001860
  03:22:32 STEP 1861 H=1861:00 +CPU= 68.617
  03:23:54 STEP 1862 H=1862:00 +CPU= 71.284
  03:26:04 STEP 1863 H=1863:00 +CPU=111.119
  03:27:25 STEP 1864 H=1864:00 +CPU= 71.107
  03:28:45 STEP 1865 H=1865:00 +CPU= 68.588
  03:31:02 STEP 1866 H=1866:00 +CPU=114.726
  03:32:22 STEP 1867 H=1867:00 +CPU= 68.611
  03:33:44 STEP 1868 H=1868:00 +CPU= 71.481
  03:35:55 STEP 1869 H=1869:00 +CPU=111.154
  03:37:17 STEP 1870 H=1870:00 +CPU= 71.175
  03:38:38 STEP 1871 H=1871:00 +CPU= 70.964
  03:41:11 STEP 1872 H=1872:00 +CPU=129.502
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_77.zip
Uploading the intermediate file: upload_file_77.zip
  03:42:30 STEP 1873 H=1873:00 +CPU= 68.255
  03:43:51 STEP 1874 H=1874:00 +CPU= 71.064
  03:46:02 STEP 1875 H=1875:00 +CPU=111.194
  03:47:24 STEP 1876 H=1876:00 +CPU= 71.010
  03:48:43 STEP 1877 H=1877:00 +CPU= 68.404
  03:50:53 STEP 1878 H=1878:00 +CPU=114.067
  03:52:11 STEP 1879 H=1879:00 +CPU= 68.417
  03:53:34 STEP 1880 H=1880:00 +CPU= 71.388
  03:55:40 STEP 1881 H=1881:00 +CPU=110.315
  03:57:03 STEP 1882 H=1882:00 +CPU= 71.041
  03:58:21 STEP 1883 H=1883:00 +CPU= 68.665
  04:00:46 STEP 1884 H=1884:00 +CPU=124.695
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001884
  04:02:04 STEP 1885 H=1885:00 +CPU= 68.764
  04:03:28 STEP 1886 H=1886:00 +CPU= 71.476
  04:05:34 STEP 1887 H=1887:00 +CPU=111.359
  04:06:57 STEP 1888 H=1888:00 +CPU= 71.334
  04:08:16 STEP 1889 H=1889:00 +CPU= 68.356
  04:10:26 STEP 1890 H=1890:00 +CPU=114.638
  04:11:47 STEP 1891 H=1891:00 +CPU= 68.320
  04:13:09 STEP 1892 H=1892:00 +CPU= 71.259
  04:15:17 STEP 1893 H=1893:00 +CPU=111.364
  04:16:36 STEP 1894 H=1894:00 +CPU= 70.986
  04:17:56 STEP 1895 H=1895:00 +CPU= 70.914
  04:20:25 STEP 1896 H=1896:00 +CPU=130.479
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_78.zip
Uploading the intermediate file: upload_file_78.zip
  04:21:45 STEP 1897 H=1897:00 +CPU= 68.454
  04:23:08 STEP 1898 H=1898:00 +CPU= 70.651
  04:25:16 STEP 1899 H=1899:00 +CPU=111.128
  04:26:36 STEP 1900 H=1900:00 +CPU= 70.846
  04:27:55 STEP 1901 H=1901:00 +CPU= 68.275
  04:30:06 STEP 1902 H=1902:00 +CPU=113.694
  04:31:24 STEP 1903 H=1903:00 +CPU= 68.513
  04:32:46 STEP 1904 H=1904:00 +CPU= 71.146
  04:34:56 STEP 1905 H=1905:00 +CPU=111.147
  04:36:20 STEP 1906 H=1906:00 +CPU= 71.451
  04:37:40 STEP 1907 H=1907:00 +CPU= 68.668
  04:40:03 STEP 1908 H=1908:00 +CPU=124.270
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001908
  04:41:23 STEP 1909 H=1909:00 +CPU= 68.753
  04:42:46 STEP 1910 H=1910:00 +CPU= 71.396
  04:44:58 STEP 1911 H=1911:00 +CPU=111.086
  04:46:21 STEP 1912 H=1912:00 +CPU= 71.642
  04:47:40 STEP 1913 H=1913:00 +CPU= 68.536
  04:49:53 STEP 1914 H=1914:00 +CPU=114.298
  04:51:09 STEP 1915 H=1915:00 +CPU= 68.261
  04:52:29 STEP 1916 H=1916:00 +CPU= 70.863
  04:54:36 STEP 1917 H=1917:00 +CPU=111.244
  04:55:57 STEP 1918 H=1918:00 +CPU= 71.066
  04:57:18 STEP 1919 H=1919:00 +CPU= 71.196
  04:59:47 STEP 1920 H=1920:00 +CPU=130.110
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_79.zip
Uploading the intermediate file: upload_file_79.zip
  05:01:06 STEP 1921 H=1921:00 +CPU= 68.270
  05:02:29 STEP 1922 H=1922:00 +CPU= 70.392
  05:04:39 STEP 1923 H=1923:00 +CPU=111.177
  05:06:00 STEP 1924 H=1924:00 +CPU= 71.348
  05:07:17 STEP 1925 H=1925:00 +CPU= 68.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:09:52 STEP 1926 H=1926:00 +CPU=114.707
  05:11:10 STEP 1927 H=1927:00 +CPU= 68.702
  05:12:31 STEP 1928 H=1928:00 +CPU= 71.396
  05:14:41 STEP 1929 H=1929:00 +CPU=112.043
  05:16:02 STEP 1930 H=1930:00 +CPU= 71.468
  05:17:22 STEP 1931 H=1931:00 +CPU= 68.493
  05:19:45 STEP 1932 H=1932:00 +CPU=124.812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001932
  05:21:03 STEP 1933 H=1933:00 +CPU= 68.906
  05:22:24 STEP 1934 H=1934:00 +CPU= 71.347
  05:24:29 STEP 1935 H=1935:00 +CPU=111.546
  05:25:51 STEP 1936 H=1936:00 +CPU= 71.262
  05:27:08 STEP 1937 H=1937:00 +CPU= 68.299
  05:29:21 STEP 1938 H=1938:00 +CPU=114.942
  05:30:42 STEP 1939 H=1939:00 +CPU= 68.456
  05:32:04 STEP 1940 H=1940:00 +CPU= 71.066
  05:34:09 STEP 1941 H=1941:00 +CPU=110.836
  05:35:32 STEP 1942 H=1942:00 +CPU= 70.849
  05:36:52 STEP 1943 H=1943:00 +CPU= 70.753
  05:39:21 STEP 1944 H=1944:00 +CPU=130.018
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_80.zip
Uploading the intermediate file: upload_file_80.zip
  05:40:41 STEP 1945 H=1945:00 +CPU= 68.625
  05:42:02 STEP 1946 H=1946:00 +CPU= 71.544
  05:44:10 STEP 1947 H=1947:00 +CPU=111.502
  05:45:31 STEP 1948 H=1948:00 +CPU= 71.432
  05:46:53 STEP 1949 H=1949:00 +CPU= 68.376
  05:49:06 STEP 1950 H=1950:00 +CPU=114.820
  05:50:26 STEP 1951 H=1951:00 +CPU= 68.570
  05:51:50 STEP 1952 H=1952:00 +CPU= 71.430
  05:54:02 STEP 1953 H=1953:00 +CPU=111.192
  05:55:24 STEP 1954 H=1954:00 +CPU= 71.326
  05:56:44 STEP 1955 H=1955:00 +CPU= 68.694
  05:59:06 STEP 1956 H=1956:00 +CPU=124.033
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001956
  06:00:26 STEP 1957 H=1957:00 +CPU= 68.772
  06:01:50 STEP 1958 H=1958:00 +CPU= 71.855
  06:04:00 STEP 1959 H=1959:00 +CPU=111.791
  06:05:22 STEP 1960 H=1960:00 +CPU= 71.344
  06:06:45 STEP 1961 H=1961:00 +CPU= 68.479
  06:08:57 STEP 1962 H=1962:00 +CPU=113.881
  06:10:12 STEP 1963 H=1963:00 +CPU= 68.233
  06:11:34 STEP 1964 H=1964:00 +CPU= 70.905
  06:13:40 STEP 1965 H=1965:00 +CPU=111.412
  06:15:02 STEP 1966 H=1966:00 +CPU= 71.752
  06:16:25 STEP 1967 H=1967:00 +CPU= 71.130
  06:18:57 STEP 1968 H=1968:00 +CPU=130.278
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_81.zip
Uploading the intermediate file: upload_file_81.zip
  06:20:18 STEP 1969 H=1969:00 +CPU= 68.637
  06:21:40 STEP 1970 H=1970:00 +CPU= 71.264
  06:23:48 STEP 1971 H=1971:00 +CPU=111.090
  06:25:09 STEP 1972 H=1972:00 +CPU= 71.513
  06:26:28 STEP 1973 H=1973:00 +CPU= 68.469
  06:28:39 STEP 1974 H=1974:00 +CPU=114.524
  06:29:55 STEP 1975 H=1975:00 +CPU= 68.726
  06:31:17 STEP 1976 H=1976:00 +CPU= 71.418
  06:33:24 STEP 1977 H=1977:00 +CPU=110.807
  06:34:45 STEP 1978 H=1978:00 +CPU= 71.821
  06:36:05 STEP 1979 H=1979:00 +CPU= 68.539
  06:38:29 STEP 1980 H=1980:00 +CPU=124.302
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001980
  06:39:49 STEP 1981 H=1981:00 +CPU= 68.686
  06:41:13 STEP 1982 H=1982:00 +CPU= 71.204
  06:43:20 STEP 1983 H=1983:00 +CPU=111.418
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:44:56 STEP 1984 H=1984:00 +CPU= 70.905
  06:46:12 STEP 1985 H=1985:00 +CPU= 68.351
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0129
(argv4) batchid: 987
(argv5) wuid: 12203773
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1e8fb4c21438175f276600405e822e57 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
ic_ancil_file: ic_ancil_12203773
ifsdata_file: ifsdata_12203773
climate_data_file: clim_data_12203773
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b616257e41b7fedeba0331227266438f to: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_203885341e62401100ca0e9a22aa1430 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203773.xml
-- Model is restarting --
Adjusting last_iter, 1986, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
last_cpu_time: 190828
upload_file_number: 82
last_iter: 1967
last_upload: 7084800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093
The child process has been launched with process id: 3008155
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3008155:3008155] [20230124:181524:1674584124:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:18:02 STEP 1968 H=1968:00 +CPU=128.053
  18:19:23 STEP 1969 H=1969:00 +CPU= 68.946
  18:20:48 STEP 1970 H=1970:00 +CPU= 71.752
  18:22:59 STEP 1971 H=1971:00 +CPU=112.493
  18:24:24 STEP 1972 H=1972:00 +CPU= 72.030
  18:25:46 STEP 1973 H=1973:00 +CPU= 69.363
  18:28:02 STEP 1974 H=1974:00 +CPU=116.027
  18:29:21 STEP 1975 H=1975:00 +CPU= 69.249
  18:30:46 STEP 1976 H=1976:00 +CPU= 73.393
  18:33:15 STEP 1977 H=1977:00 +CPU=119.364
  18:34:43 STEP 1978 H=1978:00 +CPU= 76.269
  18:36:06 STEP 1979 H=1979:00 +CPU= 72.742
  18:38:40 STEP 1980 H=1980:00 +CPU=133.284
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001980
  18:40:06 STEP 1981 H=1981:00 +CPU= 73.721
  18:41:36 STEP 1982 H=1982:00 +CPU= 76.342
  18:43:58 STEP 1983 H=1983:00 +CPU=118.809
  18:45:26 STEP 1984 H=1984:00 +CPU= 76.045
  18:46:52 STEP 1985 H=1985:00 +CPU= 73.560
  18:49:15 STEP 1986 H=1986:00 +CPU=123.002
  18:50:43 STEP 1987 H=1987:00 +CPU= 73.153
  18:52:10 STEP 1988 H=1988:00 +CPU= 76.082
  18:54:31 STEP 1989 H=1989:00 +CPU=119.180
  18:56:00 STEP 1990 H=1990:00 +CPU= 76.049
  18:57:32 STEP 1991 H=1991:00 +CPU= 76.071
  19:00:13 STEP 1992 H=1992:00 +CPU=139.173
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093_82.zip
Uploading the intermediate file: upload_file_82.zip
  19:01:36 STEP 1993 H=1993:00 +CPU= 72.670
  19:03:03 STEP 1994 H=1994:00 +CPU= 76.265
  19:05:22 STEP 1995 H=1995:00 +CPU=120.070
  19:06:52 STEP 1996 H=1996:00 +CPU= 75.856
  19:08:17 STEP 1997 H=1997:00 +CPU= 73.382
  19:10:40 STEP 1998 H=1998:00 +CPU=121.954
  19:12:08 STEP 1999 H=1999:00 +CPU= 73.346
  19:13:34 STEP 2000 H=2000:00 +CPU= 75.685
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0129
(argv4) batchid: 987
(argv5) wuid: 12203773
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203773
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1e8fb4c21438175f276600405e822e57 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0129_2018050100_123_987_12203773.zip
ic_ancil_file: ic_ancil_12203773
ifsdata_file: ifsdata_12203773
climate_data_file: clim_data_12203773
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b616257e41b7fedeba0331227266438f to: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12203773.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_203885341e62401100ca0e9a22aa1430 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12203773.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12203773.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12203773.xml
-- Model is restarting --
Adjusting last_iter, 2001, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12203773.xml
last_cpu_time: 193904
upload_file_number: 83
last_iter: 1991
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0129_2018050100_123_987_12203773_0_r44538093
The child process has been launched with process id: 3016877
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:3016877:3016877] [20230124:211306:1674594786:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:15:57 STEP 1992 H=1992:00 +CPU=135.277
  21:17:22 STEP 1993 H=1993:00 +CPU= 73.163
  21:18:51 STEP 1994 H=1994:00 +CPU= 75.905
  21:21:12 STEP 1995 H=1995:00 +CPU=118.280
  21:22:40 STEP 1996 H=1996:00 +CPU= 76.699
  21:24:07 STEP 1997 H=1997:00 +CPU= 72.894
  21:26:27 STEP 1998 H=1998:00 +CPU=121.839
  21:27:53 STEP 1999 H=1999:00 +CPU= 73.658
  21:29:21 STEP 2000 H=2000:00 +CPU= 76.035
  21:31:38 STEP 2001 H=2001:00 +CPU=117.943
  21:33:06 STEP 2002 H=2002:00 +CPU= 76.289
  21:34:33 STEP 2003 H=2003:00 +CPU= 73.662
  21:37:13 STEP 2004 H=2004:00 +CPU=134.331
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002004
  21:38:40 STEP 2005 H=2005:00 +CPU= 73.054
  21:40:11 STEP 2006 H=2006:00 +CPU= 76.557
  21:42:30 STEP 2007 H=2007:00 +CPU=118.758
  21:43:59 STEP 2008 H=2008:00 +CPU= 76.065
  21:45:23 STEP 2009 H=2009:00 +CPU= 73.302
  21:47:47 STEP 2010 H=2010:00 +CPU=122.154
  21:49:10 STEP 2011 H=2011:00 +CPU= 72.671
  21:50:38 STEP 2012 H=2012:00 +CPU= 76.255
  21:52:56 STEP 2013 H=2013:00 +CPU=119.939
  21:54:26 STEP 2014 H=2014:00 +CPU= 76.007
  21:55:57 STEP 2015 H=2015:00 +CPU= 76.031
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:35:20 1511650 22290243 oifs_43r3_ps_0129_2018050100_123_987_12203773_0 5,184,000 137,144 0.0265
09 May 2023 16:34:28 1511650 22290243 oifs_43r3_ps_0129_2018050100_123_987_12203773_0 4,320,000 113,914 0.0264
09 May 2023 16:33:40 1511650 22290243 oifs_43r3_ps_0129_2018050100_123_987_12203773_0 3,456,000 90,702 0.0262
09 May 2023 16:32:56 1511650 22290243 oifs_43r3_ps_0129_2018050100_123_987_12203773_0 2,592,000 67,434 0.0260
09 May 2023 16:32:10 1511650 22290243 oifs_43r3_ps_0129_2018050100_123_987_12203773_0 1,728,000 44,190 0.0256
09 May 2023 16:31:21 1511650 22290243 oifs_43r3_ps_0129_2018050100_123_987_12203773_0 864,000 21,341 0.0247


©2024 cpdn.org