climateprediction.net home page
Task 22246760

Task 22246760

Name oifs_43r3_ps_1672_2021050100_123_946_12164761_0
Workunit 12164761
Created 29 Nov 2022, 12:00:54 UTC
Sent 29 Nov 2022, 12:30:33 UTC
Report deadline 29 Dec 2022, 12:30:33 UTC
Received 18 Mar 2023, 7:46:32 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1524992
Run time 14 hours 30 min 47 sec
CPU time 36 min 33 sec
Validate state Invalid
Credit 11,007.36
Device peak FLOPS 6.24 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,778.53 MB
Peak swap size 4,978.08 MB
Peak disk usage 9,480.18 MB
Stderr
<core_client_version>7.22.0</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
ion.net/oifs_43r3_ps_12164761/ICMSHhpi1+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_12.zip
Uploading the intermediate file: upload_file_12.zip
  19:15:33 STEP 1825 H=1825:00 +CPU= 18.190
Uploading trickle at timestep: 6566400
  19:15:51 STEP 1826 H=1826:00 +CPU= 18.077
  19:16:19 STEP 1827 H=1827:00 +CPU= 28.385
  19:16:39 STEP 1828 H=1828:00 +CPU= 19.053
  19:16:57 STEP 1829 H=1829:00 +CPU= 18.534
  19:17:26 STEP 1830 H=1830:00 +CPU= 28.533
  19:17:44 STEP 1831 H=1831:00 +CPU= 17.482
  19:18:04 STEP 1832 H=1832:00 +CPU= 19.640
  19:18:32 STEP 1833 H=1833:00 +CPU= 28.077
  19:18:50 STEP 1834 H=1834:00 +CPU= 18.394
  19:19:09 STEP 1835 H=1835:00 +CPU= 18.140
  19:19:42 STEP 1836 H=1836:00 +CPU= 32.461
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001836
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001836
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001836
  19:19:59 STEP 1837 H=1837:00 +CPU= 17.732
  19:20:18 STEP 1838 H=1838:00 +CPU= 18.220
  19:20:48 STEP 1839 H=1839:00 +CPU= 30.006
  19:21:06 STEP 1840 H=1840:00 +CPU= 17.439
  19:21:22 STEP 1841 H=1841:00 +CPU= 16.737
  19:21:51 STEP 1842 H=1842:00 +CPU= 28.875
  19:22:09 STEP 1843 H=1843:00 +CPU= 17.668
  19:22:27 STEP 1844 H=1844:00 +CPU= 17.448
  19:22:54 STEP 1845 H=1845:00 +CPU= 26.849
  19:23:12 STEP 1846 H=1846:00 +CPU= 18.414
  19:23:30 STEP 1847 H=1847:00 +CPU= 18.028
  19:24:05 STEP 1848 H=1848:00 +CPU= 33.240
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001848
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001848
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_13.zip
  19:24:21 STEP 1849 H=1849:00 +CPU= 16.629
Uploading the intermediate file: upload_file_13.zip
  19:24:39 STEP 1850 H=1850:00 +CPU= 17.627
Uploading trickle at timestep: 6652800
  19:25:07 STEP 1851 H=1851:00 +CPU= 27.758
  19:25:27 STEP 1852 H=1852:00 +CPU= 18.722
  19:25:45 STEP 1853 H=1853:00 +CPU= 17.969
  19:26:12 STEP 1854 H=1854:00 +CPU= 27.230
  19:26:30 STEP 1855 H=1855:00 +CPU= 17.245
  19:26:48 STEP 1856 H=1856:00 +CPU= 18.189
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1672
(argv4) batchid: 946
(argv5) wuid: 12164761
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/29
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/29/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/29/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e326397d2e0b6b515c26a7c38f9c175b to: /var/lib/boinc/slots/29/oifs_43r3_ps_1672_2021050100_123_946_12164761.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/29/oifs_43r3_ps_1672_2021050100_123_946_12164761.zip
ic_ancil_file: ic_ancil_12164761
ifsdata_file: ifsdata_12164761
climate_data_file: clim_data_12164761
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_a5a6608bf2a4d34f1f97e835961a9f94 to: /var/lib/boinc/slots/29/ic_ancil_12164761.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/29/ic_ancil_12164761.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ba9c3b7f80f8fe23c5a57014c605533c to: /var/lib/boinc/slots/29/ifsdata/ifsdata_12164761.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/29/ifsdata/ifsdata_12164761.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/29/159l_2/clim_data_12164761.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/29/159l_2/clim_data_12164761.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991
The child process has been launched with process id: 272057
Executing the command: /var/lib/boinc/slots/29/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:272057:272057] [20221203:192837:1670120917:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:29:57 STEP 1848 H=1848:00 +CPU= 31.033
  19:30:15 STEP 1849 H=1849:00 +CPU= 17.193
  19:30:32 STEP 1850 H=1850:00 +CPU= 17.362
  19:30:59 STEP 1851 H=1851:00 +CPU= 26.984
  19:31:17 STEP 1852 H=1852:00 +CPU= 17.682
  19:31:35 STEP 1853 H=1853:00 +CPU= 17.948
  19:32:04 STEP 1854 H=1854:00 +CPU= 29.028
  19:32:22 STEP 1855 H=1855:00 +CPU= 17.798
  19:32:40 STEP 1856 H=1856:00 +CPU= 18.012
  19:33:08 STEP 1857 H=1857:00 +CPU= 27.857
  19:33:28 STEP 1858 H=1858:00 +CPU= 19.149
  19:33:44 STEP 1859 H=1859:00 +CPU= 16.149
  19:34:21 STEP 1860 H=1860:00 +CPU= 30.917
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001860
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001860
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001860
  19:34:39 STEP 1861 H=1861:00 +CPU= 17.822
  19:34:58 STEP 1862 H=1862:00 +CPU= 18.506
  19:35:25 STEP 1863 H=1863:00 +CPU= 27.136
  19:35:43 STEP 1864 H=1864:00 +CPU= 17.866
  19:36:01 STEP 1865 H=1865:00 +CPU= 17.415
  19:36:29 STEP 1866 H=1866:00 +CPU= 28.160
  19:36:46 STEP 1867 H=1867:00 +CPU= 17.209
  19:37:04 STEP 1868 H=1868:00 +CPU= 17.828
  19:37:32 STEP 1869 H=1869:00 +CPU= 27.240
  19:37:50 STEP 1870 H=1870:00 +CPU= 18.463
  19:38:12 STEP 1871 H=1871:00 +CPU= 17.458
  19:38:48 STEP 1872 H=1872:00 +CPU= 31.139
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001872
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001872
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_1.zip
  19:39:03 STEP 1873 H=1873:00 +CPU= 15.518
  19:39:19 STEP 1874 H=1874:00 +CPU= 15.670
Uploading the intermediate file: upload_file_1.zip
  19:39:46 STEP 1875 H=1875:00 +CPU= 26.669
Uploading trickle at timestep: 6739200
  19:40:04 STEP 1876 H=1876:00 +CPU= 17.598
  19:40:21 STEP 1877 H=1877:00 +CPU= 17.375
  19:40:49 STEP 1878 H=1878:00 +CPU= 27.843
  19:41:07 STEP 1879 H=1879:00 +CPU= 17.453
  19:41:24 STEP 1880 H=1880:00 +CPU= 17.552
  19:41:53 STEP 1881 H=1881:00 +CPU= 28.341
  19:42:11 STEP 1882 H=1882:00 +CPU= 18.205
  19:42:29 STEP 1883 H=1883:00 +CPU= 17.475
  19:42:59 STEP 1884 H=1884:00 +CPU= 30.116
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001884
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001884
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001884
  19:43:16 STEP 1885 H=1885:00 +CPU= 16.690
  19:43:35 STEP 1886 H=1886:00 +CPU= 18.712
  19:44:02 STEP 1887 H=1887:00 +CPU= 26.787
  19:44:19 STEP 1888 H=1888:00 +CPU= 17.562
  19:44:37 STEP 1889 H=1889:00 +CPU= 17.912
  19:45:07 STEP 1890 H=1890:00 +CPU= 28.947
  19:45:24 STEP 1891 H=1891:00 +CPU= 17.019
  19:45:41 STEP 1892 H=1892:00 +CPU= 17.452
  19:46:06 STEP 1893 H=1893:00 +CPU= 24.945
  19:46:23 STEP 1894 H=1894:00 +CPU= 16.620
  19:46:42 STEP 1895 H=1895:00 +CPU= 16.218
  19:48:28 STEP 1896 H=1896:00 +CPU= 31.746
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001896
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001896
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_2.zip
  19:48:46 STEP 1897 H=1897:00 +CPU= 16.673
  19:49:04 STEP 1898 H=1898:00 +CPU= 17.397
Uploading the intermediate file: upload_file_2.zip
  19:49:30 STEP 1899 H=1899:00 +CPU= 26.524
Uploading trickle at timestep: 6825600
  19:49:48 STEP 1900 H=1900:00 +CPU= 17.914
  19:50:05 STEP 1901 H=1901:00 +CPU= 16.922
  19:50:34 STEP 1902 H=1902:00 +CPU= 28.422
  19:50:51 STEP 1903 H=1903:00 +CPU= 17.632
  19:51:10 STEP 1904 H=1904:00 +CPU= 18.445
  19:51:38 STEP 1905 H=1905:00 +CPU= 27.410
  19:51:55 STEP 1906 H=1906:00 +CPU= 17.195
  19:52:13 STEP 1907 H=1907:00 +CPU= 18.112
  19:52:44 STEP 1908 H=1908:00 +CPU= 30.692
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001908
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001908
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001908
  19:53:02 STEP 1909 H=1909:00 +CPU= 17.847
  19:53:20 STEP 1910 H=1910:00 +CPU= 17.642
  19:53:47 STEP 1911 H=1911:00 +CPU= 26.860
  19:54:04 STEP 1912 H=1912:00 +CPU= 17.591
  19:54:21 STEP 1913 H=1913:00 +CPU= 16.610
  19:54:44 STEP 1914 H=1914:00 +CPU= 22.467
  19:54:59 STEP 1915 H=1915:00 +CPU= 14.954
  19:55:16 STEP 1916 H=1916:00 +CPU= 16.798
  19:55:43 STEP 1917 H=1917:00 +CPU= 27.177
  19:56:01 STEP 1918 H=1918:00 +CPU= 18.200
  19:56:19 STEP 1919 H=1919:00 +CPU= 18.055
  19:56:55 STEP 1920 H=1920:00 +CPU= 31.380
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001920
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001920
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_3.zip
Uploading the intermediate file: upload_file_3.zip
  19:57:11 STEP 1921 H=1921:00 +CPU= 15.468
  19:57:29 STEP 1922 H=1922:00 +CPU= 18.007
Uploading trickle at timestep: 6912000
  19:57:55 STEP 1923 H=1923:00 +CPU= 25.673
  19:58:12 STEP 1924 H=1924:00 +CPU= 17.539
  19:58:30 STEP 1925 H=1925:00 +CPU= 17.432
  19:58:58 STEP 1926 H=1926:00 +CPU= 28.217
  19:59:16 STEP 1927 H=1927:00 +CPU= 17.333
  19:59:33 STEP 1928 H=1928:00 +CPU= 17.298
  20:00:00 STEP 1929 H=1929:00 +CPU= 26.484
  20:00:17 STEP 1930 H=1930:00 +CPU= 17.298
  20:00:33 STEP 1931 H=1931:00 +CPU= 15.969
  20:01:02 STEP 1932 H=1932:00 +CPU= 28.487
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001932
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001932
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001932
  20:01:19 STEP 1933 H=1933:00 +CPU= 16.920
  20:01:36 STEP 1934 H=1934:00 +CPU= 17.036
  20:02:03 STEP 1935 H=1935:00 +CPU= 26.839
  20:02:21 STEP 1936 H=1936:00 +CPU= 18.243
  20:02:39 STEP 1937 H=1937:00 +CPU= 17.500
  20:03:07 STEP 1938 H=1938:00 +CPU= 27.707
  20:03:24 STEP 1939 H=1939:00 +CPU= 16.971
  20:03:42 STEP 1940 H=1940:00 +CPU= 18.086
  20:04:09 STEP 1941 H=1941:00 +CPU= 26.916
  20:04:26 STEP 1942 H=1942:00 +CPU= 17.033
  20:04:44 STEP 1943 H=1943:00 +CPU= 17.437
  20:05:16 STEP 1944 H=1944:00 +CPU= 31.337
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001944
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001944
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_4.zip
  20:05:33 STEP 1945 H=1945:00 +CPU= 16.864
Uploading the intermediate file: upload_file_4.zip
  20:05:51 STEP 1946 H=1946:00 +CPU= 17.902
Uploading trickle at timestep: 6998400
  20:06:17 STEP 1947 H=1947:00 +CPU= 26.824
  20:06:36 STEP 1948 H=1948:00 +CPU= 18.571
  20:06:52 STEP 1949 H=1949:00 +CPU= 15.532
  20:07:18 STEP 1950 H=1950:00 +CPU= 26.254
  20:07:35 STEP 1951 H=1951:00 +CPU= 17.160
  20:07:53 STEP 1952 H=1952:00 +CPU= 17.554
  20:08:19 STEP 1953 H=1953:00 +CPU= 26.366
  20:08:38 STEP 1954 H=1954:00 +CPU= 18.225
  20:08:55 STEP 1955 H=1955:00 +CPU= 16.953
  20:09:26 STEP 1956 H=1956:00 +CPU= 31.199
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001956
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001956
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001956
  20:09:46 STEP 1957 H=1957:00 +CPU= 16.791
  20:10:03 STEP 1958 H=1958:00 +CPU= 16.331
  20:10:28 STEP 1959 H=1959:00 +CPU= 24.969
  20:10:44 STEP 1960 H=1960:00 +CPU= 16.599
  20:11:01 STEP 1961 H=1961:00 +CPU= 16.442
  20:11:26 STEP 1962 H=1962:00 +CPU= 25.302
  20:11:43 STEP 1963 H=1963:00 +CPU= 16.894
  20:12:00 STEP 1964 H=1964:00 +CPU= 16.453
  20:12:25 STEP 1965 H=1965:00 +CPU= 25.431
  20:12:42 STEP 1966 H=1966:00 +CPU= 16.355
  20:12:58 STEP 1967 H=1967:00 +CPU= 16.181
  20:13:28 STEP 1968 H=1968:00 +CPU= 29.741
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001968
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001968
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_5.zip
  20:13:43 STEP 1969 H=1969:00 +CPU= 14.487
  20:13:56 STEP 1970 H=1970:00 +CPU= 13.573
Uploading the intermediate file: upload_file_5.zip
  20:14:23 STEP 1971 H=1971:00 +CPU= 26.178
Uploading trickle at timestep: 7084800
  20:14:41 STEP 1972 H=1972:00 +CPU= 18.607
  20:14:59 STEP 1973 H=1973:00 +CPU= 17.274
  20:15:28 STEP 1974 H=1974:00 +CPU= 28.704
  20:15:44 STEP 1975 H=1975:00 +CPU= 16.486
  20:16:02 STEP 1976 H=1976:00 +CPU= 17.450
  20:16:28 STEP 1977 H=1977:00 +CPU= 25.852
  20:16:46 STEP 1978 H=1978:00 +CPU= 18.223
  20:17:04 STEP 1979 H=1979:00 +CPU= 17.337
  20:17:34 STEP 1980 H=1980:00 +CPU= 30.048
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001980
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001980
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001980
  20:17:51 STEP 1981 H=1981:00 +CPU= 17.234
  20:18:09 STEP 1982 H=1982:00 +CPU= 17.463
  20:18:37 STEP 1983 H=1983:00 +CPU= 27.999
  20:18:53 STEP 1984 H=1984:00 +CPU= 16.368
  20:19:09 STEP 1985 H=1985:00 +CPU= 15.636
  20:19:36 STEP 1986 H=1986:00 +CPU= 27.088
  20:19:54 STEP 1987 H=1987:00 +CPU= 17.583
  20:20:12 STEP 1988 H=1988:00 +CPU= 18.115
  20:20:39 STEP 1989 H=1989:00 +CPU= 26.774
  20:20:56 STEP 1990 H=1990:00 +CPU= 16.645
  20:21:13 STEP 1991 H=1991:00 +CPU= 16.947
  20:21:44 STEP 1992 H=1992:00 +CPU= 31.249
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+001992
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+001992
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_6.zip
  20:22:02 STEP 1993 H=1993:00 +CPU= 17.708
  20:22:19 STEP 1994 H=1994:00 +CPU= 17.083
Uploading the intermediate file: upload_file_6.zip
  20:22:45 STEP 1995 H=1995:00 +CPU= 25.331
Uploading trickle at timestep: 7171200
  20:23:01 STEP 1996 H=1996:00 +CPU= 15.985
  20:23:17 STEP 1997 H=1997:00 +CPU= 15.960
  20:23:45 STEP 1998 H=1998:00 +CPU= 27.343
  20:24:00 STEP 1999 H=1999:00 +CPU= 14.879
  20:24:17 STEP 2000 H=2000:00 +CPU= 16.482
  20:24:45 STEP 2001 H=2001:00 +CPU= 27.556
  20:25:02 STEP 2002 H=2002:00 +CPU= 17.413
  20:25:20 STEP 2003 H=2003:00 +CPU= 17.563
  20:25:51 STEP 2004 H=2004:00 +CPU= 31.078
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002004
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002004
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002004
  20:26:08 STEP 2005 H=2005:00 +CPU= 17.143
  20:26:26 STEP 2006 H=2006:00 +CPU= 17.955
  20:26:55 STEP 2007 H=2007:00 +CPU= 28.241
  20:27:13 STEP 2008 H=2008:00 +CPU= 18.375
  20:27:32 STEP 2009 H=2009:00 +CPU= 18.659
  20:28:00 STEP 2010 H=2010:00 +CPU= 28.384
  20:28:16 STEP 2011 H=2011:00 +CPU= 15.691
  20:28:34 STEP 2012 H=2012:00 +CPU= 17.741
  20:29:01 STEP 2013 H=2013:00 +CPU= 27.375
  20:29:20 STEP 2014 H=2014:00 +CPU= 18.171
  20:29:39 STEP 2015 H=2015:00 +CPU= 17.860
  20:30:29 STEP 2016 H=2016:00 +CPU= 31.607
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002016
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002016
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_7.zip
  20:30:47 STEP 2017 H=2017:00 +CPU= 17.118
Uploading the intermediate file: upload_file_7.zip
  20:31:05 STEP 2018 H=2018:00 +CPU= 17.045
Uploading trickle at timestep: 7257600
  20:31:32 STEP 2019 H=2019:00 +CPU= 26.874
  20:31:48 STEP 2020 H=2020:00 +CPU= 16.543
  20:32:04 STEP 2021 H=2021:00 +CPU= 15.339
  20:32:29 STEP 2022 H=2022:00 +CPU= 24.669
  20:32:46 STEP 2023 H=2023:00 +CPU= 17.243
  20:33:05 STEP 2024 H=2024:00 +CPU= 18.520
  20:33:32 STEP 2025 H=2025:00 +CPU= 27.238
  20:33:50 STEP 2026 H=2026:00 +CPU= 17.693
  20:34:08 STEP 2027 H=2027:00 +CPU= 17.343
  20:34:39 STEP 2028 H=2028:00 +CPU= 30.806
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002028
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002028
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002028
  20:34:56 STEP 2029 H=2029:00 +CPU= 17.366
  20:35:15 STEP 2030 H=2030:00 +CPU= 18.128
  20:35:43 STEP 2031 H=2031:00 +CPU= 27.743
  20:36:01 STEP 2032 H=2032:00 +CPU= 18.096
  20:36:19 STEP 2033 H=2033:00 +CPU= 17.632
  20:36:48 STEP 2034 H=2034:00 +CPU= 28.539
  20:37:05 STEP 2035 H=2035:00 +CPU= 17.179
  20:37:23 STEP 2036 H=2036:00 +CPU= 17.768
  20:37:50 STEP 2037 H=2037:00 +CPU= 26.825
  20:38:07 STEP 2038 H=2038:00 +CPU= 17.044
  20:38:28 STEP 2039 H=2039:00 +CPU= 18.166
  20:39:45 STEP 2040 H=2040:00 +CPU= 31.179
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002040
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002040
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_8.zip
  20:40:02 STEP 2041 H=2041:00 +CPU= 17.050
Uploading the intermediate file: upload_file_8.zip
  20:40:21 STEP 2042 H=2042:00 +CPU= 18.064
Uploading trickle at timestep: 7344000
  20:40:48 STEP 2043 H=2043:00 +CPU= 27.636
  20:41:07 STEP 2044 H=2044:00 +CPU= 18.641
  20:41:24 STEP 2045 H=2045:00 +CPU= 17.186
  20:41:54 STEP 2046 H=2046:00 +CPU= 29.387
  20:42:11 STEP 2047 H=2047:00 +CPU= 17.175
  20:42:29 STEP 2048 H=2048:00 +CPU= 17.655
  20:42:57 STEP 2049 H=2049:00 +CPU= 28.337
  20:43:16 STEP 2050 H=2050:00 +CPU= 18.405
  20:43:33 STEP 2051 H=2051:00 +CPU= 16.873
  20:44:05 STEP 2052 H=2052:00 +CPU= 31.731
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002052
  20:44:23 STEP 2053 H=2053:00 +CPU= 17.858
  20:44:42 STEP 2054 H=2054:00 +CPU= 18.346
  20:45:09 STEP 2055 H=2055:00 +CPU= 26.811
  20:45:26 STEP 2056 H=2056:00 +CPU= 17.133
  20:45:43 STEP 2057 H=2057:00 +CPU= 16.272
  20:46:10 STEP 2058 H=2058:00 +CPU= 27.621
  20:46:27 STEP 2059 H=2059:00 +CPU= 16.764
  20:46:44 STEP 2060 H=2060:00 +CPU= 16.843
  20:47:12 STEP 2061 H=2061:00 +CPU= 27.920
  20:47:31 STEP 2062 H=2062:00 +CPU= 18.177
  20:47:49 STEP 2063 H=2063:00 +CPU= 17.758
  20:48:25 STEP 2064 H=2064:00 +CPU= 32.636
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002064
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002064
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_9.zip
  20:48:43 STEP 2065 H=2065:00 +CPU= 17.288
Uploading the intermediate file: upload_file_9.zip
  20:49:01 STEP 2066 H=2066:00 +CPU= 17.489
Uploading trickle at timestep: 7430400
  20:49:25 STEP 2067 H=2067:00 +CPU= 24.070
  20:49:44 STEP 2068 H=2068:00 +CPU= 18.674
  20:50:03 STEP 2069 H=2069:00 +CPU= 17.946
  20:50:33 STEP 2070 H=2070:00 +CPU= 29.400
  20:50:51 STEP 2071 H=2071:00 +CPU= 17.832
  20:51:11 STEP 2072 H=2072:00 +CPU= 19.411
  20:51:39 STEP 2073 H=2073:00 +CPU= 28.372
  20:51:55 STEP 2074 H=2074:00 +CPU= 15.628
  20:52:10 STEP 2075 H=2075:00 +CPU= 14.778
  20:52:41 STEP 2076 H=2076:00 +CPU= 31.255
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002076
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002076
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002076
  20:53:00 STEP 2077 H=2077:00 +CPU= 18.286
  20:53:18 STEP 2078 H=2078:00 +CPU= 17.935
  20:53:45 STEP 2079 H=2079:00 +CPU= 27.047
  20:54:03 STEP 2080 H=2080:00 +CPU= 18.290
  20:54:20 STEP 2081 H=2081:00 +CPU= 16.901
  20:54:49 STEP 2082 H=2082:00 +CPU= 28.698
  20:55:07 STEP 2083 H=2083:00 +CPU= 17.370
  20:55:24 STEP 2084 H=2084:00 +CPU= 17.386
  20:55:52 STEP 2085 H=2085:00 +CPU= 27.266
  20:56:10 STEP 2086 H=2086:00 +CPU= 18.248
  20:56:29 STEP 2087 H=2087:00 +CPU= 17.910
  20:57:23 STEP 2088 H=2088:00 +CPU= 32.225
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002088
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002088
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_10.zip
  20:57:41 STEP 2089 H=2089:00 +CPU= 17.728
Uploading the intermediate file: upload_file_10.zip
  20:57:56 STEP 2090 H=2090:00 +CPU= 14.870
Uploading trickle at timestep: 7516800
  20:58:24 STEP 2091 H=2091:00 +CPU= 27.637
  20:58:41 STEP 2092 H=2092:00 +CPU= 17.816
  20:58:59 STEP 2093 H=2093:00 +CPU= 17.676
  20:59:27 STEP 2094 H=2094:00 +CPU= 27.459
  20:59:45 STEP 2095 H=2095:00 +CPU= 17.081
  21:00:03 STEP 2096 H=2096:00 +CPU= 18.053
  21:00:30 STEP 2097 H=2097:00 +CPU= 26.778
  21:00:49 STEP 2098 H=2098:00 +CPU= 19.401
  21:01:06 STEP 2099 H=2099:00 +CPU= 16.620
  21:01:35 STEP 2100 H=2100:00 +CPU= 28.610
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002100
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002100
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002100
  21:01:54 STEP 2101 H=2101:00 +CPU= 18.522
  21:02:12 STEP 2102 H=2102:00 +CPU= 18.093
  21:02:40 STEP 2103 H=2103:00 +CPU= 28.385
  21:02:59 STEP 2104 H=2104:00 +CPU= 18.035
  21:03:16 STEP 2105 H=2105:00 +CPU= 17.147
  21:03:45 STEP 2106 H=2106:00 +CPU= 28.963
  21:04:02 STEP 2107 H=2107:00 +CPU= 17.217
  21:04:20 STEP 2108 H=2108:00 +CPU= 18.018
  21:04:47 STEP 2109 H=2109:00 +CPU= 26.811
  21:05:04 STEP 2110 H=2110:00 +CPU= 16.652
  21:05:25 STEP 2111 H=2111:00 +CPU= 18.397
  21:06:12 STEP 2112 H=2112:00 +CPU= 32.126
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002112
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002112
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_11.zip
  21:06:30 STEP 2113 H=2113:00 +CPU= 17.378
Uploading the intermediate file: upload_file_11.zip
  21:06:46 STEP 2114 H=2114:00 +CPU= 15.544
Uploading trickle at timestep: 7603200
  21:07:12 STEP 2115 H=2115:00 +CPU= 26.328
  21:07:31 STEP 2116 H=2116:00 +CPU= 18.962
  21:07:49 STEP 2117 H=2117:00 +CPU= 17.649
  21:08:18 STEP 2118 H=2118:00 +CPU= 28.330
  21:08:36 STEP 2119 H=2119:00 +CPU= 17.448
  21:08:53 STEP 2120 H=2120:00 +CPU= 17.230
  21:09:20 STEP 2121 H=2121:00 +CPU= 26.863
  21:09:38 STEP 2122 H=2122:00 +CPU= 17.471
  21:09:55 STEP 2123 H=2123:00 +CPU= 17.410
  21:10:26 STEP 2124 H=2124:00 +CPU= 30.543
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002124
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002124
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002124
  21:10:44 STEP 2125 H=2125:00 +CPU= 17.700
  21:11:01 STEP 2126 H=2126:00 +CPU= 17.295
  21:11:29 STEP 2127 H=2127:00 +CPU= 27.453
  21:11:47 STEP 2128 H=2128:00 +CPU= 18.584
  21:12:06 STEP 2129 H=2129:00 +CPU= 18.486
  21:12:34 STEP 2130 H=2130:00 +CPU= 28.299
  21:12:52 STEP 2131 H=2131:00 +CPU= 17.091
  21:13:09 STEP 2132 H=2132:00 +CPU= 17.296
  21:13:37 STEP 2133 H=2133:00 +CPU= 27.433
  21:13:54 STEP 2134 H=2134:00 +CPU= 17.415
  21:14:13 STEP 2135 H=2135:00 +CPU= 18.263
  21:14:47 STEP 2136 H=2136:00 +CPU= 32.209
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002136
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002136
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_12.zip
Uploading the intermediate file: upload_file_12.zip
  21:15:05 STEP 2137 H=2137:00 +CPU= 18.434
  21:15:23 STEP 2138 H=2138:00 +CPU= 17.943
Uploading trickle at timestep: 7689600
  21:15:51 STEP 2139 H=2139:00 +CPU= 27.664
  21:16:09 STEP 2140 H=2140:00 +CPU= 17.335
  21:16:27 STEP 2141 H=2141:00 +CPU= 17.545
  21:16:56 STEP 2142 H=2142:00 +CPU= 29.219
  21:17:14 STEP 2143 H=2143:00 +CPU= 18.065
  21:17:34 STEP 2144 H=2144:00 +CPU= 19.148
  21:18:01 STEP 2145 H=2145:00 +CPU= 27.055
  21:18:19 STEP 2146 H=2146:00 +CPU= 17.902
  21:18:37 STEP 2147 H=2147:00 +CPU= 17.517
  21:19:07 STEP 2148 H=2148:00 +CPU= 30.102
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002148
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002148
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002148
  21:19:25 STEP 2149 H=2149:00 +CPU= 17.264
  21:19:42 STEP 2150 H=2150:00 +CPU= 17.293
  21:20:10 STEP 2151 H=2151:00 +CPU= 28.175
  21:20:30 STEP 2152 H=2152:00 +CPU= 19.244
  21:20:47 STEP 2153 H=2153:00 +CPU= 17.289
  21:21:16 STEP 2154 H=2154:00 +CPU= 29.075
  21:21:34 STEP 2155 H=2155:00 +CPU= 17.389
  21:21:52 STEP 2156 H=2156:00 +CPU= 17.583
  21:22:19 STEP 2157 H=2157:00 +CPU= 27.562
  21:22:38 STEP 2158 H=2158:00 +CPU= 18.642
  21:22:57 STEP 2159 H=2159:00 +CPU= 18.752
  21:23:33 STEP 2160 H=2160:00 +CPU= 32.845
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002160
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002160
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_13.zip
Uploading the intermediate file: upload_file_13.zip
  21:23:52 STEP 2161 H=2161:00 +CPU= 18.791
  21:24:10 STEP 2162 H=2162:00 +CPU= 17.441
Uploading trickle at timestep: 7776000
  21:24:37 STEP 2163 H=2163:00 +CPU= 27.629
  21:24:56 STEP 2164 H=2164:00 +CPU= 18.688
  21:25:13 STEP 2165 H=2165:00 +CPU= 16.819
  21:25:41 STEP 2166 H=2166:00 +CPU= 27.452
  21:25:57 STEP 2167 H=2167:00 +CPU= 15.895
  21:26:15 STEP 2168 H=2168:00 +CPU= 17.545
  21:26:43 STEP 2169 H=2169:00 +CPU= 27.451
  21:27:01 STEP 2170 H=2170:00 +CPU= 17.706
  21:27:19 STEP 2171 H=2171:00 +CPU= 17.159
  21:27:51 STEP 2172 H=2172:00 +CPU= 31.917
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002172
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1672
(argv4) batchid: 946
(argv5) wuid: 12164761
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/29
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/29/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/29/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e326397d2e0b6b515c26a7c38f9c175b to: /var/lib/boinc/slots/29/oifs_43r3_ps_1672_2021050100_123_946_12164761.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/29/oifs_43r3_ps_1672_2021050100_123_946_12164761.zip
ic_ancil_file: ic_ancil_12164761
ifsdata_file: ifsdata_12164761
climate_data_file: clim_data_12164761
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_a5a6608bf2a4d34f1f97e835961a9f94 to: /var/lib/boinc/slots/29/ic_ancil_12164761.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/29/ic_ancil_12164761.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ba9c3b7f80f8fe23c5a57014c605533c to: /var/lib/boinc/slots/29/ifsdata/ifsdata_12164761.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/29/ifsdata/ifsdata_12164761.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/29/159l_2/clim_data_12164761.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/29/159l_2/clim_data_12164761.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991
The child process has been launched with process id: 275301
Executing the command: /var/lib/boinc/slots/29/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:guoj-desktop:1:1:275301:275301] [20221203:212915:1670128155:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:29:49 STEP 2160 H=2160:00 +CPU= 31.191
  21:30:03 STEP 2161 H=2161:00 +CPU= 13.573
  21:30:20 STEP 2162 H=2162:00 +CPU= 16.281
  21:30:48 STEP 2163 H=2163:00 +CPU= 27.428
  21:31:07 STEP 2164 H=2164:00 +CPU= 18.060
  21:31:25 STEP 2165 H=2165:00 +CPU= 17.138
  21:31:54 STEP 2166 H=2166:00 +CPU= 27.342
  21:32:11 STEP 2167 H=2167:00 +CPU= 16.869
  21:32:28 STEP 2168 H=2168:00 +CPU= 17.664
  21:32:56 STEP 2169 H=2169:00 +CPU= 27.158
  21:33:14 STEP 2170 H=2170:00 +CPU= 17.692
  21:33:31 STEP 2171 H=2171:00 +CPU= 16.636
  21:34:01 STEP 2172 H=2172:00 +CPU= 29.689
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002172
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002172
  21:34:18 STEP 2173 H=2173:00 +CPU= 16.882
  21:34:36 STEP 2174 H=2174:00 +CPU= 17.135
  21:35:04 STEP 2175 H=2175:00 +CPU= 27.402
  21:35:22 STEP 2176 H=2176:00 +CPU= 16.896
  21:35:39 STEP 2177 H=2177:00 +CPU= 16.935
  21:36:09 STEP 2178 H=2178:00 +CPU= 28.157
  21:36:27 STEP 2179 H=2179:00 +CPU= 17.453
  21:36:45 STEP 2180 H=2180:00 +CPU= 17.816
  21:37:14 STEP 2181 H=2181:00 +CPU= 28.062
  21:37:33 STEP 2182 H=2182:00 +CPU= 18.250
  21:37:56 STEP 2183 H=2183:00 +CPU= 17.689
  21:40:08 STEP 2184 H=2184:00 +CPU= 31.395
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002184
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002184
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_1.zip
Uploading the intermediate file: upload_file_1.zip
  21:40:24 STEP 2185 H=2185:00 +CPU= 15.104
  21:40:40 STEP 2186 H=2186:00 +CPU= 16.344
Uploading trickle at timestep: 7862400
  21:41:07 STEP 2187 H=2187:00 +CPU= 26.980
  21:41:25 STEP 2188 H=2188:00 +CPU= 17.213
  21:41:41 STEP 2189 H=2189:00 +CPU= 16.401
  21:42:10 STEP 2190 H=2190:00 +CPU= 27.773
  21:42:26 STEP 2191 H=2191:00 +CPU= 16.070
  21:42:44 STEP 2192 H=2192:00 +CPU= 17.599
  21:43:11 STEP 2193 H=2193:00 +CPU= 26.419
  21:43:30 STEP 2194 H=2194:00 +CPU= 17.866
  21:43:47 STEP 2195 H=2195:00 +CPU= 16.470
  21:44:16 STEP 2196 H=2196:00 +CPU= 28.747
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002196
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002196
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002196
  21:44:33 STEP 2197 H=2197:00 +CPU= 16.451
  21:44:51 STEP 2198 H=2198:00 +CPU= 17.481
  21:45:17 STEP 2199 H=2199:00 +CPU= 25.784
  21:45:34 STEP 2200 H=2200:00 +CPU= 16.965
  21:45:51 STEP 2201 H=2201:00 +CPU= 16.556
  21:46:18 STEP 2202 H=2202:00 +CPU= 26.396
  21:46:34 STEP 2203 H=2203:00 +CPU= 16.350
  21:46:51 STEP 2204 H=2204:00 +CPU= 16.896
  21:47:18 STEP 2205 H=2205:00 +CPU= 26.118
  21:47:34 STEP 2206 H=2206:00 +CPU= 16.193
  21:47:51 STEP 2207 H=2207:00 +CPU= 16.266
  21:49:12 STEP 2208 H=2208:00 +CPU= 31.071
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002208
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002208
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_2.zip
  21:49:27 STEP 2209 H=2209:00 +CPU= 15.745
Uploading the intermediate file: upload_file_2.zip
  21:49:45 STEP 2210 H=2210:00 +CPU= 17.280
Uploading trickle at timestep: 7948800
  21:50:10 STEP 2211 H=2211:00 +CPU= 25.150
  21:50:29 STEP 2212 H=2212:00 +CPU= 18.245
  21:50:48 STEP 2213 H=2213:00 +CPU= 16.876
  21:51:14 STEP 2214 H=2214:00 +CPU= 26.406
  21:51:33 STEP 2215 H=2215:00 +CPU= 18.115
  21:51:50 STEP 2216 H=2216:00 +CPU= 17.234
  21:52:16 STEP 2217 H=2217:00 +CPU= 25.341
  21:52:33 STEP 2218 H=2218:00 +CPU= 16.967
  21:52:49 STEP 2219 H=2219:00 +CPU= 16.429
  21:53:16 STEP 2220 H=2220:00 +CPU= 26.609
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002220
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002220
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002220
  21:53:32 STEP 2221 H=2221:00 +CPU= 15.794
  21:53:49 STEP 2222 H=2222:00 +CPU= 16.976
  21:54:16 STEP 2223 H=2223:00 +CPU= 27.029
  21:54:34 STEP 2224 H=2224:00 +CPU= 17.763
  21:54:52 STEP 2225 H=2225:00 +CPU= 17.221
  21:55:20 STEP 2226 H=2226:00 +CPU= 28.150
  21:55:38 STEP 2227 H=2227:00 +CPU= 17.162
  21:55:54 STEP 2228 H=2228:00 +CPU= 15.842
  21:56:16 STEP 2229 H=2229:00 +CPU= 22.266
  21:56:33 STEP 2230 H=2230:00 +CPU= 16.800
  21:56:51 STEP 2231 H=2231:00 +CPU= 17.692
  21:57:26 STEP 2232 H=2232:00 +CPU= 31.512
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002232
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002232
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002220
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_3.zip
  21:57:43 STEP 2233 H=2233:00 +CPU= 17.451
Uploading the intermediate file: upload_file_3.zip
  21:58:01 STEP 2234 H=2234:00 +CPU= 17.732
Uploading trickle at timestep: 8035200
  21:58:29 STEP 2235 H=2235:00 +CPU= 27.398
  21:58:47 STEP 2236 H=2236:00 +CPU= 18.115
  21:59:05 STEP 2237 H=2237:00 +CPU= 17.666
  21:59:31 STEP 2238 H=2238:00 +CPU= 25.274
  21:59:49 STEP 2239 H=2239:00 +CPU= 17.832
  22:00:05 STEP 2240 H=2240:00 +CPU= 16.192
  22:00:32 STEP 2241 H=2241:00 +CPU= 25.959
  22:00:51 STEP 2242 H=2242:00 +CPU= 18.497
  22:01:08 STEP 2243 H=2243:00 +CPU= 17.373
  22:01:38 STEP 2244 H=2244:00 +CPU= 29.548
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002244
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002244
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002244
  22:01:55 STEP 2245 H=2245:00 +CPU= 16.315
  22:02:12 STEP 2246 H=2246:00 +CPU= 17.122
  22:02:37 STEP 2247 H=2247:00 +CPU= 24.709
  22:02:55 STEP 2248 H=2248:00 +CPU= 17.545
  22:03:12 STEP 2249 H=2249:00 +CPU= 16.882
  22:03:39 STEP 2250 H=2250:00 +CPU= 27.274
  22:03:56 STEP 2251 H=2251:00 +CPU= 16.411
  22:04:13 STEP 2252 H=2252:00 +CPU= 16.830
  22:04:39 STEP 2253 H=2253:00 +CPU= 26.282
  22:04:57 STEP 2254 H=2254:00 +CPU= 17.376
  22:05:15 STEP 2255 H=2255:00 +CPU= 17.823
  22:05:48 STEP 2256 H=2256:00 +CPU= 32.567
Moving to projects directory: /var/lib/boinc/slots/29/ICMGGhpi1+002256
Moving to projects directory: /var/lib/boinc/slots/29/ICMSHhpi1+002256
Moving to projects directory: /var/lib/boinc/slots/29/ICMUAhpi1+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMGGhpi1+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMSHhpi1+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761/ICMUAhpi1+002244
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_1672_2021050100_123_946_12164761_0_r1943471991_4.zip
Uploading the intermediate file: upload_file_4.zip
  22:06:05 STEP 2257 H=2257:00 +CPU= 16.409
Uploading trickle at timestep: 8121600
  22:06:23 STEP 2258 H=2258:00 +CPU= 17.979
  22:06:51 STEP 2259 H=2259:00 +CPU= 27.280
  22:07:10 STEP 2260 H=2260:00 +CPU= 18.726
  22:07:28 STEP 2261 H=2261:00 +CPU= 17.930
  22:07:56 STEP 2262 H=2262:00 +CPU= 27.862
  22:08:13 STEP 2263 H=2263:00 +CPU= 17.082
  22:08:32 STEP 2264 H=2264:00 +CPU= 18.576
  22:08:59 STEP 2265 H=2265:00 +CPU= 27.205
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1672
(argv4) batchid: 946
(argv5) wuid: 12164761
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/29
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12164761
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/29/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/29/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e326397d2e0b6b515c26a7c38f9c175b to: /var/lib/boinc/slots/29/oifs_43r3_ps_1672_2021050100_123_946_12164761.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/29/oifs_43r3_ps_1672_2021050100_123_946_12164761.zip
ic_ancil_file: ic_ancil_12164761
ifsdata_file: ifsdata_12164761
climate_data_file: clim_data_12164761
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_a5a6608bf2a4d34f1f97e835961a9f94 to: /var/lib/boinc/slots/29/ic_ancil_12164761.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/29/ic_ancil_12164761.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ba9c3b7f80f8fe23c5a57014c605533c to: /var/lib/boinc/slots/29/ifsdata/ifsdata_12164761.zip

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:32:53 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 8,121,600 1,999 0.0002
09 May 2023 15:32:53 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 8,035,200 1,503 0.0002
09 May 2023 15:32:53 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,948,800 1,023 0.0001
09 May 2023 15:32:53 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,862,400 535 0.0001
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,776,000 6,589 0.0008
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,689,600 6,069 0.0008
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,603,200 5,563 0.0007
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,516,800 5,050 0.0007
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,430,400 4,544 0.0006
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,344,000 4,034 0.0005
09 May 2023 15:32:46 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,257,600 3,523 0.0005
09 May 2023 15:32:36 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,171,200 3,024 0.0004
09 May 2023 15:32:36 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 7,084,800 2,536 0.0004
09 May 2023 15:32:37 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,998,400 2,049 0.0003
09 May 2023 15:32:37 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,912,000 1,545 0.0002
09 May 2023 15:32:36 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,825,600 1,051 0.0002
09 May 2023 15:32:36 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,739,200 550 0.0001
09 May 2023 15:32:36 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,652,800 6,695 0.0010
09 May 2023 15:32:29 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,566,400 6,168 0.0009
09 May 2023 15:32:29 1524992 22246760 oifs_43r3_ps_1672_2021050100_123_946_12164761_0 6,480,000 5,643 0.0009


©2024 climateprediction.net