climateprediction.net home page
Task 22246725

Task 22246725

Name oifs_43r3_ps_1637_2021050100_123_946_12164726_0
Workunit 12164726
Created 29 Nov 2022, 11:55:33 UTC
Sent 29 Nov 2022, 12:30:33 UTC
Report deadline 29 Dec 2022, 12:30:33 UTC
Received 18 Mar 2023, 7:46:32 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1524992
Run time 14 hours 29 min 44 sec
CPU time 37 min 14 sec
Validate state Invalid
Credit 11,249.28
Device peak FLOPS 6.24 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,779.27 MB
Peak swap size 4,978.08 MB
Peak disk usage 9,479.84 MB
Stderr
<core_client_version>7.22.0</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
83_12.zip
Uploading the intermediate file: upload_file_12.zip
  19:14:59 STEP 1825 H=1825:00 +CPU= 16.964
Uploading trickle at timestep: 6566400
  19:15:18 STEP 1826 H=1826:00 +CPU= 18.862
  19:15:47 STEP 1827 H=1827:00 +CPU= 28.177
  19:16:05 STEP 1828 H=1828:00 +CPU= 18.259
  19:16:23 STEP 1829 H=1829:00 +CPU= 17.584
  19:16:52 STEP 1830 H=1830:00 +CPU= 29.222
  19:17:10 STEP 1831 H=1831:00 +CPU= 17.936
  19:17:29 STEP 1832 H=1832:00 +CPU= 18.730
  19:17:57 STEP 1833 H=1833:00 +CPU= 27.601
  19:18:15 STEP 1834 H=1834:00 +CPU= 17.767
  19:18:33 STEP 1835 H=1835:00 +CPU= 18.340
  19:19:06 STEP 1836 H=1836:00 +CPU= 32.019
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001836
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001836
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001836
  19:19:24 STEP 1837 H=1837:00 +CPU= 18.501
  19:19:44 STEP 1838 H=1838:00 +CPU= 19.574
  19:20:10 STEP 1839 H=1839:00 +CPU= 26.142
  19:20:29 STEP 1840 H=1840:00 +CPU= 18.491
  19:20:48 STEP 1841 H=1841:00 +CPU= 19.118
  19:21:17 STEP 1842 H=1842:00 +CPU= 28.240
  19:21:35 STEP 1843 H=1843:00 +CPU= 18.282
  19:21:53 STEP 1844 H=1844:00 +CPU= 17.536
  19:22:21 STEP 1845 H=1845:00 +CPU= 28.240
  19:22:38 STEP 1846 H=1846:00 +CPU= 17.387
  19:22:57 STEP 1847 H=1847:00 +CPU= 18.064
  19:23:30 STEP 1848 H=1848:00 +CPU= 33.192
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001848
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001848
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_13.zip
  19:23:49 STEP 1849 H=1849:00 +CPU= 18.293
Uploading the intermediate file: upload_file_13.zip
  19:24:07 STEP 1850 H=1850:00 +CPU= 18.333
Uploading trickle at timestep: 6652800
  19:24:34 STEP 1851 H=1851:00 +CPU= 26.861
  19:24:53 STEP 1852 H=1852:00 +CPU= 18.151
  19:25:10 STEP 1853 H=1853:00 +CPU= 17.727
  19:25:38 STEP 1854 H=1854:00 +CPU= 27.171
  19:25:55 STEP 1855 H=1855:00 +CPU= 17.004
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1637
(argv4) batchid: 946
(argv5) wuid: 12164726
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/34
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/34/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/34/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5d3fb0dae1c92a384dfd25bf7abbb83b to: /var/lib/boinc/slots/34/oifs_43r3_ps_1637_2021050100_123_946_12164726.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/34/oifs_43r3_ps_1637_2021050100_123_946_12164726.zip
ic_ancil_file: ic_ancil_12164726
ifsdata_file: ifsdata_12164726
climate_data_file: clim_data_12164726
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_6536e9571c94503e1d63b5bf9813f917 to: /var/lib/boinc/slots/34/ic_ancil_12164726.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/34/ic_ancil_12164726.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_94ec420ccf2b3e7a7b441a3f3eec70b0 to: /var/lib/boinc/slots/34/ifsdata/ifsdata_12164726.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/34/ifsdata/ifsdata_12164726.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/34/159l_2/clim_data_12164726.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/34/159l_2/clim_data_12164726.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583
The child process has been launched with process id: 272037
Executing the command: /var/lib/boinc/slots/34/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272037:272037] [20221203:192748:1670120868:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:28:43 STEP 1848 H=1848:00 +CPU= 31.137
  19:29:02 STEP 1849 H=1849:00 +CPU= 17.942
  19:29:17 STEP 1850 H=1850:00 +CPU= 14.873
  19:29:44 STEP 1851 H=1851:00 +CPU= 27.005
  19:30:03 STEP 1852 H=1852:00 +CPU= 18.428
  19:30:20 STEP 1853 H=1853:00 +CPU= 17.615
  19:30:49 STEP 1854 H=1854:00 +CPU= 27.917
  19:31:07 STEP 1855 H=1855:00 +CPU= 18.136
  19:31:26 STEP 1856 H=1856:00 +CPU= 18.798
  19:31:53 STEP 1857 H=1857:00 +CPU= 27.502
  19:32:12 STEP 1858 H=1858:00 +CPU= 18.604
  19:32:30 STEP 1859 H=1859:00 +CPU= 17.547
  19:33:01 STEP 1860 H=1860:00 +CPU= 30.549
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001860
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001860
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001860
  19:33:19 STEP 1861 H=1861:00 +CPU= 17.872
  19:33:37 STEP 1862 H=1862:00 +CPU= 18.681
  19:34:04 STEP 1863 H=1863:00 +CPU= 25.740
  19:34:23 STEP 1864 H=1864:00 +CPU= 18.971
  19:34:42 STEP 1865 H=1865:00 +CPU= 18.765
  19:35:10 STEP 1866 H=1866:00 +CPU= 27.864
  19:35:26 STEP 1867 H=1867:00 +CPU= 16.752
  19:35:45 STEP 1868 H=1868:00 +CPU= 18.195
  19:36:12 STEP 1869 H=1869:00 +CPU= 26.986
  19:36:29 STEP 1870 H=1870:00 +CPU= 16.627
  19:36:47 STEP 1871 H=1871:00 +CPU= 17.464
  19:37:22 STEP 1872 H=1872:00 +CPU= 30.871
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001872
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001872
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_1.zip
  19:37:39 STEP 1873 H=1873:00 +CPU= 17.400
  19:37:58 STEP 1874 H=1874:00 +CPU= 18.306
Uploading the intermediate file: upload_file_1.zip
  19:38:23 STEP 1875 H=1875:00 +CPU= 25.077
Uploading trickle at timestep: 6739200
  19:38:41 STEP 1876 H=1876:00 +CPU= 17.540
  19:38:57 STEP 1877 H=1877:00 +CPU= 16.165
  19:39:24 STEP 1878 H=1878:00 +CPU= 27.438
  19:39:43 STEP 1879 H=1879:00 +CPU= 18.300
  19:40:01 STEP 1880 H=1880:00 +CPU= 17.956
  19:40:29 STEP 1881 H=1881:00 +CPU= 27.623
  19:40:47 STEP 1882 H=1882:00 +CPU= 17.764
  19:41:04 STEP 1883 H=1883:00 +CPU= 17.439
  19:41:35 STEP 1884 H=1884:00 +CPU= 30.742
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001884
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001884
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001884
  19:41:53 STEP 1885 H=1885:00 +CPU= 17.298
  19:42:11 STEP 1886 H=1886:00 +CPU= 18.271
  19:42:38 STEP 1887 H=1887:00 +CPU= 26.430
  19:42:56 STEP 1888 H=1888:00 +CPU= 18.154
  19:43:13 STEP 1889 H=1889:00 +CPU= 17.271
  19:43:42 STEP 1890 H=1890:00 +CPU= 28.297
  19:44:00 STEP 1891 H=1891:00 +CPU= 17.381
  19:44:17 STEP 1892 H=1892:00 +CPU= 17.483
  19:44:45 STEP 1893 H=1893:00 +CPU= 28.121
  19:45:03 STEP 1894 H=1894:00 +CPU= 17.462
  19:45:22 STEP 1895 H=1895:00 +CPU= 18.958
  19:45:53 STEP 1896 H=1896:00 +CPU= 29.448
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001896
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001896
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_2.zip
Uploading the intermediate file: upload_file_2.zip
  19:46:08 STEP 1897 H=1897:00 +CPU= 15.699
Uploading trickle at timestep: 6825600
  19:46:24 STEP 1898 H=1898:00 +CPU= 15.295
  19:46:50 STEP 1899 H=1899:00 +CPU= 26.412
  19:47:07 STEP 1900 H=1900:00 +CPU= 15.914
  19:47:23 STEP 1901 H=1901:00 +CPU= 16.175
  19:47:50 STEP 1902 H=1902:00 +CPU= 26.510
  19:48:07 STEP 1903 H=1903:00 +CPU= 16.994
  19:48:23 STEP 1904 H=1904:00 +CPU= 16.450
  19:48:51 STEP 1905 H=1905:00 +CPU= 27.364
  19:49:08 STEP 1906 H=1906:00 +CPU= 16.679
  19:49:23 STEP 1907 H=1907:00 +CPU= 15.388
  19:49:53 STEP 1908 H=1908:00 +CPU= 29.972
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001908
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001908
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001908
  19:50:11 STEP 1909 H=1909:00 +CPU= 17.749
  19:50:29 STEP 1910 H=1910:00 +CPU= 17.598
  19:50:57 STEP 1911 H=1911:00 +CPU= 28.290
  19:51:17 STEP 1912 H=1912:00 +CPU= 19.020
  19:51:34 STEP 1913 H=1913:00 +CPU= 17.388
  19:52:03 STEP 1914 H=1914:00 +CPU= 28.412
  19:52:21 STEP 1915 H=1915:00 +CPU= 18.076
  19:52:39 STEP 1916 H=1916:00 +CPU= 18.447
  19:53:06 STEP 1917 H=1917:00 +CPU= 26.868
  19:53:25 STEP 1918 H=1918:00 +CPU= 18.111
  19:53:42 STEP 1919 H=1919:00 +CPU= 17.358
  19:54:18 STEP 1920 H=1920:00 +CPU= 32.448
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001920
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001920
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_3.zip
  19:54:32 STEP 1921 H=1921:00 +CPU= 14.250
  19:54:48 STEP 1922 H=1922:00 +CPU= 15.838
Uploading the intermediate file: upload_file_3.zip
  19:55:15 STEP 1923 H=1923:00 +CPU= 26.942
Uploading trickle at timestep: 6912000
  19:55:34 STEP 1924 H=1924:00 +CPU= 18.544
  19:55:52 STEP 1925 H=1925:00 +CPU= 18.173
  19:56:19 STEP 1926 H=1926:00 +CPU= 27.132
  19:56:36 STEP 1927 H=1927:00 +CPU= 17.160
  19:56:54 STEP 1928 H=1928:00 +CPU= 17.890
  19:57:20 STEP 1929 H=1929:00 +CPU= 25.497
  19:57:37 STEP 1930 H=1930:00 +CPU= 16.760
  19:57:53 STEP 1931 H=1931:00 +CPU= 16.111
  19:58:22 STEP 1932 H=1932:00 +CPU= 28.968
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001932
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001932
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001932
  19:58:39 STEP 1933 H=1933:00 +CPU= 16.262
  19:58:56 STEP 1934 H=1934:00 +CPU= 17.604
  19:59:22 STEP 1935 H=1935:00 +CPU= 25.667
  19:59:40 STEP 1936 H=1936:00 +CPU= 17.113
  19:59:57 STEP 1937 H=1937:00 +CPU= 17.169
  20:00:23 STEP 1938 H=1938:00 +CPU= 26.200
  20:00:39 STEP 1939 H=1939:00 +CPU= 15.913
  20:00:57 STEP 1940 H=1940:00 +CPU= 17.414
  20:01:24 STEP 1941 H=1941:00 +CPU= 27.467
  20:01:41 STEP 1942 H=1942:00 +CPU= 16.743
  20:02:00 STEP 1943 H=1943:00 +CPU= 18.287
  20:02:35 STEP 1944 H=1944:00 +CPU= 33.325
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001944
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001944
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_4.zip
  20:02:52 STEP 1945 H=1945:00 +CPU= 17.064
Uploading the intermediate file: upload_file_4.zip
  20:03:08 STEP 1946 H=1946:00 +CPU= 16.243
Uploading trickle at timestep: 6998400
  20:03:36 STEP 1947 H=1947:00 +CPU= 27.596
  20:03:54 STEP 1948 H=1948:00 +CPU= 18.035
  20:04:12 STEP 1949 H=1949:00 +CPU= 17.433
  20:04:39 STEP 1950 H=1950:00 +CPU= 27.665
  20:04:57 STEP 1951 H=1951:00 +CPU= 17.618
  20:05:15 STEP 1952 H=1952:00 +CPU= 17.521
  20:05:42 STEP 1953 H=1953:00 +CPU= 26.787
  20:05:59 STEP 1954 H=1954:00 +CPU= 17.090
  20:06:16 STEP 1955 H=1955:00 +CPU= 16.144
  20:06:47 STEP 1956 H=1956:00 +CPU= 31.173
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001956
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001956
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001956
  20:07:03 STEP 1957 H=1957:00 +CPU= 16.481
  20:07:21 STEP 1958 H=1958:00 +CPU= 17.268
  20:07:48 STEP 1959 H=1959:00 +CPU= 26.972
  20:08:06 STEP 1960 H=1960:00 +CPU= 17.793
  20:08:22 STEP 1961 H=1961:00 +CPU= 15.784
  20:08:48 STEP 1962 H=1962:00 +CPU= 26.785
  20:09:06 STEP 1963 H=1963:00 +CPU= 17.444
  20:09:24 STEP 1964 H=1964:00 +CPU= 17.707
  20:09:48 STEP 1965 H=1965:00 +CPU= 24.281
  20:10:04 STEP 1966 H=1966:00 +CPU= 16.019
  20:10:22 STEP 1967 H=1967:00 +CPU= 16.241
  20:11:01 STEP 1968 H=1968:00 +CPU= 31.753
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001968
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001968
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_5.zip
  20:11:17 STEP 1969 H=1969:00 +CPU= 16.205
Uploading the intermediate file: upload_file_5.zip
  20:11:33 STEP 1970 H=1970:00 +CPU= 16.250
Uploading trickle at timestep: 7084800
  20:12:00 STEP 1971 H=1971:00 +CPU= 26.417
  20:12:18 STEP 1972 H=1972:00 +CPU= 18.026
  20:12:33 STEP 1973 H=1973:00 +CPU= 15.386
  20:13:00 STEP 1974 H=1974:00 +CPU= 26.399
  20:13:16 STEP 1975 H=1975:00 +CPU= 15.855
  20:13:33 STEP 1976 H=1976:00 +CPU= 16.670
  20:13:55 STEP 1977 H=1977:00 +CPU= 22.185
  20:14:11 STEP 1978 H=1978:00 +CPU= 16.384
  20:14:29 STEP 1979 H=1979:00 +CPU= 17.011
  20:15:01 STEP 1980 H=1980:00 +CPU= 31.482
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001980
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001980
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001980
  20:15:19 STEP 1981 H=1981:00 +CPU= 17.934
  20:15:37 STEP 1982 H=1982:00 +CPU= 17.839
  20:16:05 STEP 1983 H=1983:00 +CPU= 28.077
  20:16:23 STEP 1984 H=1984:00 +CPU= 17.091
  20:16:40 STEP 1985 H=1985:00 +CPU= 16.967
  20:17:07 STEP 1986 H=1986:00 +CPU= 27.564
  20:17:25 STEP 1987 H=1987:00 +CPU= 17.198
  20:17:42 STEP 1988 H=1988:00 +CPU= 16.981
  20:18:09 STEP 1989 H=1989:00 +CPU= 27.513
  20:18:27 STEP 1990 H=1990:00 +CPU= 17.441
  20:18:46 STEP 1991 H=1991:00 +CPU= 18.162
  20:19:19 STEP 1992 H=1992:00 +CPU= 31.701
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+001992
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+001992
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_6.zip
  20:19:35 STEP 1993 H=1993:00 +CPU= 16.605
Uploading the intermediate file: upload_file_6.zip
  20:19:54 STEP 1994 H=1994:00 +CPU= 18.027
Uploading trickle at timestep: 7171200
  20:20:22 STEP 1995 H=1995:00 +CPU= 27.990
  20:20:40 STEP 1996 H=1996:00 +CPU= 17.876
  20:20:57 STEP 1997 H=1997:00 +CPU= 17.357
  20:21:23 STEP 1998 H=1998:00 +CPU= 25.813
  20:21:41 STEP 1999 H=1999:00 +CPU= 17.392
  20:21:59 STEP 2000 H=2000:00 +CPU= 18.257
  20:22:26 STEP 2001 H=2001:00 +CPU= 26.375
  20:22:42 STEP 2002 H=2002:00 +CPU= 16.270
  20:23:00 STEP 2003 H=2003:00 +CPU= 17.207
  20:23:30 STEP 2004 H=2004:00 +CPU= 29.913
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002004
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002004
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002004
  20:23:48 STEP 2005 H=2005:00 +CPU= 17.285
  20:24:05 STEP 2006 H=2006:00 +CPU= 16.984
  20:24:33 STEP 2007 H=2007:00 +CPU= 27.593
  20:24:50 STEP 2008 H=2008:00 +CPU= 17.900
  20:25:07 STEP 2009 H=2009:00 +CPU= 16.921
  20:25:36 STEP 2010 H=2010:00 +CPU= 28.515
  20:25:54 STEP 2011 H=2011:00 +CPU= 17.913
  20:26:11 STEP 2012 H=2012:00 +CPU= 16.658
  20:26:37 STEP 2013 H=2013:00 +CPU= 25.938
  20:26:54 STEP 2014 H=2014:00 +CPU= 16.533
  20:27:11 STEP 2015 H=2015:00 +CPU= 16.764
  20:27:44 STEP 2016 H=2016:00 +CPU= 31.944
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002016
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002016
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_7.zip
  20:28:02 STEP 2017 H=2017:00 +CPU= 17.852
Uploading the intermediate file: upload_file_7.zip
  20:28:18 STEP 2018 H=2018:00 +CPU= 15.262
Uploading trickle at timestep: 7257600
  20:28:45 STEP 2019 H=2019:00 +CPU= 26.892
  20:29:02 STEP 2020 H=2020:00 +CPU= 17.593
  20:29:20 STEP 2021 H=2021:00 +CPU= 17.612
  20:29:49 STEP 2022 H=2022:00 +CPU= 29.171
  20:30:05 STEP 2023 H=2023:00 +CPU= 15.415
  20:30:23 STEP 2024 H=2024:00 +CPU= 17.634
  20:30:51 STEP 2025 H=2025:00 +CPU= 26.877
  20:31:09 STEP 2026 H=2026:00 +CPU= 17.666
  20:31:26 STEP 2027 H=2027:00 +CPU= 17.536
  20:31:54 STEP 2028 H=2028:00 +CPU= 27.575
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002028
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002028
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002028
  20:32:11 STEP 2029 H=2029:00 +CPU= 16.577
  20:32:26 STEP 2030 H=2030:00 +CPU= 15.065
  20:32:54 STEP 2031 H=2031:00 +CPU= 27.328
  20:33:10 STEP 2032 H=2032:00 +CPU= 16.523
  20:33:28 STEP 2033 H=2033:00 +CPU= 17.347
  20:33:57 STEP 2034 H=2034:00 +CPU= 28.674
  20:34:14 STEP 2035 H=2035:00 +CPU= 17.230
  20:34:32 STEP 2036 H=2036:00 +CPU= 18.166
  20:35:00 STEP 2037 H=2037:00 +CPU= 27.531
  20:35:18 STEP 2038 H=2038:00 +CPU= 18.185
  20:35:37 STEP 2039 H=2039:00 +CPU= 18.275
  20:36:17 STEP 2040 H=2040:00 +CPU= 32.645
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002040
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002040
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_8.zip
Uploading the intermediate file: upload_file_8.zip
  20:36:35 STEP 2041 H=2041:00 +CPU= 18.106
  20:36:53 STEP 2042 H=2042:00 +CPU= 17.550
Uploading trickle at timestep: 7344000
  20:37:21 STEP 2043 H=2043:00 +CPU= 27.274
  20:37:39 STEP 2044 H=2044:00 +CPU= 18.416
  20:37:56 STEP 2045 H=2045:00 +CPU= 17.058
  20:38:25 STEP 2046 H=2046:00 +CPU= 28.657
  20:38:42 STEP 2047 H=2047:00 +CPU= 17.252
  20:38:59 STEP 2048 H=2048:00 +CPU= 16.179
  20:39:24 STEP 2049 H=2049:00 +CPU= 25.184
  20:39:41 STEP 2050 H=2050:00 +CPU= 16.705
  20:39:57 STEP 2051 H=2051:00 +CPU= 15.935
  20:40:27 STEP 2052 H=2052:00 +CPU= 29.478
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002052
  20:40:45 STEP 2053 H=2053:00 +CPU= 17.894
  20:41:03 STEP 2054 H=2054:00 +CPU= 17.920
  20:41:31 STEP 2055 H=2055:00 +CPU= 27.884
  20:41:51 STEP 2056 H=2056:00 +CPU= 19.188
  20:42:08 STEP 2057 H=2057:00 +CPU= 17.544
  20:42:36 STEP 2058 H=2058:00 +CPU= 27.352
  20:42:53 STEP 2059 H=2059:00 +CPU= 17.183
  20:43:12 STEP 2060 H=2060:00 +CPU= 18.506
  20:43:40 STEP 2061 H=2061:00 +CPU= 28.349
  20:44:00 STEP 2062 H=2062:00 +CPU= 18.523
  20:44:19 STEP 2063 H=2063:00 +CPU= 17.274
  20:45:13 STEP 2064 H=2064:00 +CPU= 31.903
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002064
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002064
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_9.zip
  20:45:30 STEP 2065 H=2065:00 +CPU= 17.045
Uploading the intermediate file: upload_file_9.zip
  20:45:48 STEP 2066 H=2066:00 +CPU= 17.573
Uploading trickle at timestep: 7430400
  20:46:15 STEP 2067 H=2067:00 +CPU= 26.606
  20:46:32 STEP 2068 H=2068:00 +CPU= 17.538
  20:46:49 STEP 2069 H=2069:00 +CPU= 17.001
  20:47:18 STEP 2070 H=2070:00 +CPU= 28.890
  20:47:36 STEP 2071 H=2071:00 +CPU= 17.465
  20:47:55 STEP 2072 H=2072:00 +CPU= 18.430
  20:48:22 STEP 2073 H=2073:00 +CPU= 27.776
  20:48:40 STEP 2074 H=2074:00 +CPU= 17.626
  20:48:58 STEP 2075 H=2075:00 +CPU= 17.241
  20:49:25 STEP 2076 H=2076:00 +CPU= 27.577
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002076
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002076
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002076
  20:49:43 STEP 2077 H=2077:00 +CPU= 17.756
  20:50:02 STEP 2078 H=2078:00 +CPU= 18.948
  20:50:31 STEP 2079 H=2079:00 +CPU= 28.458
  20:50:50 STEP 2080 H=2080:00 +CPU= 18.722
  20:51:09 STEP 2081 H=2081:00 +CPU= 18.071
  20:51:38 STEP 2082 H=2082:00 +CPU= 28.891
  20:51:54 STEP 2083 H=2083:00 +CPU= 15.844
  20:52:09 STEP 2084 H=2084:00 +CPU= 15.357
  20:52:37 STEP 2085 H=2085:00 +CPU= 27.554
  20:52:54 STEP 2086 H=2086:00 +CPU= 17.300
  20:53:14 STEP 2087 H=2087:00 +CPU= 17.612
  20:54:08 STEP 2088 H=2088:00 +CPU= 31.788
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002088
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002088
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_10.zip
Uploading the intermediate file: upload_file_10.zip
  20:54:26 STEP 2089 H=2089:00 +CPU= 17.369
Uploading trickle at timestep: 7516800
  20:54:44 STEP 2090 H=2090:00 +CPU= 17.888
  20:55:12 STEP 2091 H=2091:00 +CPU= 28.081
  20:55:29 STEP 2092 H=2092:00 +CPU= 17.085
  20:55:47 STEP 2093 H=2093:00 +CPU= 17.816
  20:56:15 STEP 2094 H=2094:00 +CPU= 27.663
  20:56:33 STEP 2095 H=2095:00 +CPU= 17.733
  20:56:51 STEP 2096 H=2096:00 +CPU= 18.286
  20:57:19 STEP 2097 H=2097:00 +CPU= 27.876
  20:57:37 STEP 2098 H=2098:00 +CPU= 17.907
  20:57:52 STEP 2099 H=2099:00 +CPU= 14.974
  20:58:22 STEP 2100 H=2100:00 +CPU= 29.516
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002100
  20:58:40 STEP 2101 H=2101:00 +CPU= 17.606
  20:58:58 STEP 2102 H=2102:00 +CPU= 18.639
  20:59:26 STEP 2103 H=2103:00 +CPU= 27.042
  20:59:44 STEP 2104 H=2104:00 +CPU= 17.736
  21:00:02 STEP 2105 H=2105:00 +CPU= 17.777
  21:00:29 STEP 2106 H=2106:00 +CPU= 27.550
  21:00:46 STEP 2107 H=2107:00 +CPU= 16.874
  21:01:05 STEP 2108 H=2108:00 +CPU= 18.359
  21:01:31 STEP 2109 H=2109:00 +CPU= 25.774
  21:01:50 STEP 2110 H=2110:00 +CPU= 19.073
  21:02:09 STEP 2111 H=2111:00 +CPU= 18.335
  21:02:48 STEP 2112 H=2112:00 +CPU= 33.563
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_11.zip
Uploading the intermediate file: upload_file_11.zip
  21:03:05 STEP 2113 H=2113:00 +CPU= 17.209
Uploading trickle at timestep: 7603200
  21:03:23 STEP 2114 H=2114:00 +CPU= 17.044
  21:03:50 STEP 2115 H=2115:00 +CPU= 27.300
  21:04:09 STEP 2116 H=2116:00 +CPU= 18.775
  21:04:27 STEP 2117 H=2117:00 +CPU= 18.239
  21:04:54 STEP 2118 H=2118:00 +CPU= 26.744
  21:05:11 STEP 2119 H=2119:00 +CPU= 16.400
  21:05:29 STEP 2120 H=2120:00 +CPU= 18.191
  21:05:56 STEP 2121 H=2121:00 +CPU= 27.028
  21:06:16 STEP 2122 H=2122:00 +CPU= 18.923
  21:06:33 STEP 2123 H=2123:00 +CPU= 17.547
  21:07:01 STEP 2124 H=2124:00 +CPU= 27.229
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002124
  21:07:19 STEP 2125 H=2125:00 +CPU= 17.945
  21:07:38 STEP 2126 H=2126:00 +CPU= 18.923
  21:08:06 STEP 2127 H=2127:00 +CPU= 27.583
  21:08:25 STEP 2128 H=2128:00 +CPU= 18.761
  21:08:42 STEP 2129 H=2129:00 +CPU= 17.159
  21:09:09 STEP 2130 H=2130:00 +CPU= 27.455
  21:09:27 STEP 2131 H=2131:00 +CPU= 17.790
  21:09:45 STEP 2132 H=2132:00 +CPU= 17.264
  21:10:12 STEP 2133 H=2133:00 +CPU= 27.060
  21:10:30 STEP 2134 H=2134:00 +CPU= 18.200
  21:10:49 STEP 2135 H=2135:00 +CPU= 17.404
  21:11:25 STEP 2136 H=2136:00 +CPU= 32.854
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_12.zip
  21:11:44 STEP 2137 H=2137:00 +CPU= 18.162
Uploading the intermediate file: upload_file_12.zip
  21:12:03 STEP 2138 H=2138:00 +CPU= 18.934
Uploading trickle at timestep: 7689600
  21:12:31 STEP 2139 H=2139:00 +CPU= 27.786
  21:12:48 STEP 2140 H=2140:00 +CPU= 17.658
  21:13:06 STEP 2141 H=2141:00 +CPU= 17.308
  21:13:34 STEP 2142 H=2142:00 +CPU= 28.044
  21:13:51 STEP 2143 H=2143:00 +CPU= 17.053
  21:14:10 STEP 2144 H=2144:00 +CPU= 18.670
  21:14:37 STEP 2145 H=2145:00 +CPU= 27.000
  21:14:55 STEP 2146 H=2146:00 +CPU= 17.940
  21:15:13 STEP 2147 H=2147:00 +CPU= 17.848
  21:15:45 STEP 2148 H=2148:00 +CPU= 31.049
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002148
  21:16:01 STEP 2149 H=2149:00 +CPU= 15.933
  21:16:19 STEP 2150 H=2150:00 +CPU= 17.922
  21:16:47 STEP 2151 H=2151:00 +CPU= 27.705
  21:17:04 STEP 2152 H=2152:00 +CPU= 17.254
  21:17:22 STEP 2153 H=2153:00 +CPU= 17.931
  21:17:51 STEP 2154 H=2154:00 +CPU= 28.035
  21:18:08 STEP 2155 H=2155:00 +CPU= 17.594
  21:18:25 STEP 2156 H=2156:00 +CPU= 16.403
  21:18:53 STEP 2157 H=2157:00 +CPU= 27.554
  21:19:11 STEP 2158 H=2158:00 +CPU= 17.865
  21:19:28 STEP 2159 H=2159:00 +CPU= 17.439
  21:20:11 STEP 2160 H=2160:00 +CPU= 32.608
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_13.zip
  21:20:29 STEP 2161 H=2161:00 +CPU= 18.211
Uploading the intermediate file: upload_file_13.zip
  21:20:48 STEP 2162 H=2162:00 +CPU= 18.229
Uploading trickle at timestep: 7776000
  21:21:16 STEP 2163 H=2163:00 +CPU= 28.081
  21:21:33 STEP 2164 H=2164:00 +CPU= 17.131
  21:21:50 STEP 2165 H=2165:00 +CPU= 16.557
  21:22:19 STEP 2166 H=2166:00 +CPU= 28.863
  21:22:37 STEP 2167 H=2167:00 +CPU= 18.051
  21:22:56 STEP 2168 H=2168:00 +CPU= 18.605
  21:23:23 STEP 2169 H=2169:00 +CPU= 27.099
  21:23:43 STEP 2170 H=2170:00 +CPU= 19.355
  21:24:02 STEP 2171 H=2171:00 +CPU= 18.845
  21:24:33 STEP 2172 H=2172:00 +CPU= 30.839
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002172
  21:24:50 STEP 2173 H=2173:00 +CPU= 17.435
  21:25:09 STEP 2174 H=2174:00 +CPU= 18.284
  21:25:36 STEP 2175 H=2175:00 +CPU= 26.940
  21:25:53 STEP 2176 H=2176:00 +CPU= 16.753
  21:26:10 STEP 2177 H=2177:00 +CPU= 16.689
  21:26:39 STEP 2178 H=2178:00 +CPU= 28.838
  21:26:57 STEP 2179 H=2179:00 +CPU= 17.034
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1637
(argv4) batchid: 946
(argv5) wuid: 12164726
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/34
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/34/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/34/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5d3fb0dae1c92a384dfd25bf7abbb83b to: /var/lib/boinc/slots/34/oifs_43r3_ps_1637_2021050100_123_946_12164726.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/34/oifs_43r3_ps_1637_2021050100_123_946_12164726.zip
ic_ancil_file: ic_ancil_12164726
ifsdata_file: ifsdata_12164726
climate_data_file: clim_data_12164726
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_6536e9571c94503e1d63b5bf9813f917 to: /var/lib/boinc/slots/34/ic_ancil_12164726.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/34/ic_ancil_12164726.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_94ec420ccf2b3e7a7b441a3f3eec70b0 to: /var/lib/boinc/slots/34/ifsdata/ifsdata_12164726.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/34/ifsdata/ifsdata_12164726.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/34/159l_2/clim_data_12164726.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/34/159l_2/clim_data_12164726.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583
The child process has been launched with process id: 275296
Executing the command: /var/lib/boinc/slots/34/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275296:275296] [20221203:212906:1670128146:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:29:40 STEP 2160 H=2160:00 +CPU= 31.086
  21:29:56 STEP 2161 H=2161:00 +CPU= 16.268
  21:30:11 STEP 2162 H=2162:00 +CPU= 15.006
  21:30:39 STEP 2163 H=2163:00 +CPU= 26.988
  21:30:57 STEP 2164 H=2164:00 +CPU= 18.115
  21:31:15 STEP 2165 H=2165:00 +CPU= 17.128
  21:31:44 STEP 2166 H=2166:00 +CPU= 28.162
  21:32:02 STEP 2167 H=2167:00 +CPU= 17.248
  21:32:20 STEP 2168 H=2168:00 +CPU= 17.877
  21:32:47 STEP 2169 H=2169:00 +CPU= 26.712
  21:33:05 STEP 2170 H=2170:00 +CPU= 17.781
  21:33:23 STEP 2171 H=2171:00 +CPU= 17.297
  21:33:53 STEP 2172 H=2172:00 +CPU= 29.937
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002172
  21:34:10 STEP 2173 H=2173:00 +CPU= 17.029
  21:34:28 STEP 2174 H=2174:00 +CPU= 17.860
  21:34:57 STEP 2175 H=2175:00 +CPU= 27.079
  21:35:14 STEP 2176 H=2176:00 +CPU= 17.562
  21:35:32 STEP 2177 H=2177:00 +CPU= 17.033
  21:36:02 STEP 2178 H=2178:00 +CPU= 28.079
  21:36:19 STEP 2179 H=2179:00 +CPU= 17.261
  21:36:38 STEP 2180 H=2180:00 +CPU= 17.872
  21:37:07 STEP 2181 H=2181:00 +CPU= 27.758
  21:37:24 STEP 2182 H=2182:00 +CPU= 17.671
  21:37:45 STEP 2183 H=2183:00 +CPU= 18.444
  21:39:55 STEP 2184 H=2184:00 +CPU= 31.285
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002184
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002184
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_1.zip
Uploading the intermediate file: upload_file_1.zip
  21:40:12 STEP 2185 H=2185:00 +CPU= 16.120
  21:40:29 STEP 2186 H=2186:00 +CPU= 16.403
Uploading trickle at timestep: 7862400
  21:40:55 STEP 2187 H=2187:00 +CPU= 25.807
  21:41:12 STEP 2188 H=2188:00 +CPU= 17.536
  21:41:29 STEP 2189 H=2189:00 +CPU= 16.686
  21:41:58 STEP 2190 H=2190:00 +CPU= 28.546
  21:42:16 STEP 2191 H=2191:00 +CPU= 17.381
  21:42:35 STEP 2192 H=2192:00 +CPU= 17.654
  21:43:02 STEP 2193 H=2193:00 +CPU= 27.016
  21:43:20 STEP 2194 H=2194:00 +CPU= 17.395
  21:43:38 STEP 2195 H=2195:00 +CPU= 17.786
  21:44:08 STEP 2196 H=2196:00 +CPU= 29.303
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002196
  21:44:25 STEP 2197 H=2197:00 +CPU= 16.260
  21:44:43 STEP 2198 H=2198:00 +CPU= 18.095
  21:45:09 STEP 2199 H=2199:00 +CPU= 26.190
  21:45:25 STEP 2200 H=2200:00 +CPU= 15.575
  21:45:42 STEP 2201 H=2201:00 +CPU= 16.665
  21:46:09 STEP 2202 H=2202:00 +CPU= 26.445
  21:46:25 STEP 2203 H=2203:00 +CPU= 16.053
  21:46:42 STEP 2204 H=2204:00 +CPU= 16.922
  21:47:08 STEP 2205 H=2205:00 +CPU= 25.164
  21:47:26 STEP 2206 H=2206:00 +CPU= 17.679
  21:47:44 STEP 2207 H=2207:00 +CPU= 17.076
  21:48:38 STEP 2208 H=2208:00 +CPU= 30.908
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002208
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002208
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_2.zip
Uploading the intermediate file: upload_file_2.zip
  21:48:54 STEP 2209 H=2209:00 +CPU= 16.124
Uploading trickle at timestep: 7948800
  21:49:11 STEP 2210 H=2210:00 +CPU= 16.890
  21:49:38 STEP 2211 H=2211:00 +CPU= 26.573
  21:49:56 STEP 2212 H=2212:00 +CPU= 17.868
  21:50:13 STEP 2213 H=2213:00 +CPU= 17.046
  21:50:42 STEP 2214 H=2214:00 +CPU= 28.496
  21:51:01 STEP 2215 H=2215:00 +CPU= 17.490
  21:51:19 STEP 2216 H=2216:00 +CPU= 18.040
  21:51:47 STEP 2217 H=2217:00 +CPU= 27.925
  21:52:05 STEP 2218 H=2218:00 +CPU= 17.165
  21:52:19 STEP 2219 H=2219:00 +CPU= 14.470
  21:52:48 STEP 2220 H=2220:00 +CPU= 27.919
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002220
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002220
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002220
  21:53:03 STEP 2221 H=2221:00 +CPU= 15.075
  21:53:20 STEP 2222 H=2222:00 +CPU= 17.279
  21:53:48 STEP 2223 H=2223:00 +CPU= 27.429
  21:54:05 STEP 2224 H=2224:00 +CPU= 16.821
  21:54:23 STEP 2225 H=2225:00 +CPU= 18.424
  21:54:51 STEP 2226 H=2226:00 +CPU= 27.574
  21:55:08 STEP 2227 H=2227:00 +CPU= 17.185
  21:55:26 STEP 2228 H=2228:00 +CPU= 17.831
  21:55:53 STEP 2229 H=2229:00 +CPU= 26.129
  21:56:06 STEP 2230 H=2230:00 +CPU= 13.591
  21:56:22 STEP 2231 H=2231:00 +CPU= 15.477
  21:57:02 STEP 2232 H=2232:00 +CPU= 32.482
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002232
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002232
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002220
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_3.zip
  21:57:20 STEP 2233 H=2233:00 +CPU= 16.835
Uploading the intermediate file: upload_file_3.zip
  21:57:37 STEP 2234 H=2234:00 +CPU= 17.216
Uploading trickle at timestep: 8035200
  21:58:04 STEP 2235 H=2235:00 +CPU= 26.994
  21:58:22 STEP 2236 H=2236:00 +CPU= 17.928
  21:58:39 STEP 2237 H=2237:00 +CPU= 16.832
  21:59:08 STEP 2238 H=2238:00 +CPU= 28.362
  21:59:24 STEP 2239 H=2239:00 +CPU= 15.885
  21:59:39 STEP 2240 H=2240:00 +CPU= 14.842
  22:00:06 STEP 2241 H=2241:00 +CPU= 26.767
  22:00:23 STEP 2242 H=2242:00 +CPU= 16.961
  22:00:41 STEP 2243 H=2243:00 +CPU= 17.533
  22:01:13 STEP 2244 H=2244:00 +CPU= 31.378
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002244
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002244
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002244
  22:01:30 STEP 2245 H=2245:00 +CPU= 16.521
  22:01:48 STEP 2246 H=2246:00 +CPU= 17.996
  22:02:16 STEP 2247 H=2247:00 +CPU= 27.396
  22:02:32 STEP 2248 H=2248:00 +CPU= 16.032
  22:02:49 STEP 2249 H=2249:00 +CPU= 16.949
  22:03:16 STEP 2250 H=2250:00 +CPU= 27.033
  22:03:34 STEP 2251 H=2251:00 +CPU= 16.857
  22:03:51 STEP 2252 H=2252:00 +CPU= 16.921
  22:04:18 STEP 2253 H=2253:00 +CPU= 26.566
  22:04:35 STEP 2254 H=2254:00 +CPU= 17.348
  22:04:53 STEP 2255 H=2255:00 +CPU= 17.629
  22:05:25 STEP 2256 H=2256:00 +CPU= 31.858
Moving to projects directory: /var/lib/boinc/slots/34/ICMGGhpi1+002256
Moving to projects directory: /var/lib/boinc/slots/34/ICMSHhpi1+002256
Moving to projects directory: /var/lib/boinc/slots/34/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMGGhpi1+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMSHhpi1+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726/ICMUAhpi1+002244
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1637_2021050100_123_946_12164726_0_r100354583_4.zip
Uploading the intermediate file: upload_file_4.zip
  22:05:44 STEP 2257 H=2257:00 +CPU= 18.001
Uploading trickle at timestep: 8121600
  22:06:01 STEP 2258 H=2258:00 +CPU= 17.538
  22:06:30 STEP 2259 H=2259:00 +CPU= 27.997
  22:06:48 STEP 2260 H=2260:00 +CPU= 17.877
  22:07:06 STEP 2261 H=2261:00 +CPU= 17.601
  22:07:35 STEP 2262 H=2262:00 +CPU= 28.681
  22:07:53 STEP 2263 H=2263:00 +CPU= 17.498
  22:08:11 STEP 2264 H=2264:00 +CPU= 17.926
  22:08:40 STEP 2265 H=2265:00 +CPU= 28.773
  22:08:58 STEP 2266 H=2266:00 +CPU= 17.961
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1637
(argv4) batchid: 946
(argv5) wuid: 12164726
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/34
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164726
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/34/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/34/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5d3fb0dae1c92a384dfd25bf7abbb83b to: /var/lib/boinc/slots/34/oifs_43r3_ps_1637_2021050100_123_946_12164726.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/34/oifs_43r3_ps_1637_2021050100_123_946_12164726.zip
ic_ancil_file: ic_ancil_12164726
ifsdata_file: ifsdata_12164726
climate_data_file: clim_data_12164726
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_6536e9571c94503e1d63b5bf9813f917 to: /var/lib/boinc/slots/34/ic_ancil_12164726.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/34/ic_ancil_12164726.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_94ec420ccf2b3e7a7b441a3f3eec70b0 to: /var/lib/boinc/slots/34/ifsdata/ifsdata_12164726.zip

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:32:53 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 8,121,600 2,021 0.0002
09 May 2023 15:32:54 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 8,035,200 1,518 0.0002
09 May 2023 15:32:54 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,948,800 1,035 0.0001
09 May 2023 15:32:53 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,862,400 544 0.0001
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,776,000 6,579 0.0008
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,689,600 6,063 0.0008
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,603,200 5,560 0.0007
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,516,800 5,043 0.0007
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,430,400 4,544 0.0006
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,344,000 4,029 0.0005
09 May 2023 15:32:46 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,257,600 3,533 0.0005
09 May 2023 15:32:37 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,171,200 3,034 0.0004
09 May 2023 15:32:37 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 7,084,800 2,540 0.0004
09 May 2023 15:32:36 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,998,400 2,040 0.0003
09 May 2023 15:32:36 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,912,000 1,548 0.0002
09 May 2023 15:32:36 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,825,600 1,048 0.0002
09 May 2023 15:32:37 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,739,200 543 0.0001
09 May 2023 15:32:37 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,652,800 6,700 0.0010
09 May 2023 15:32:29 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,566,400 6,174 0.0009
09 May 2023 15:32:29 1524992 22246725 oifs_43r3_ps_1637_2021050100_123_946_12164726_0 6,480,000 5,659 0.0009


©2024 climateprediction.net