climateprediction.net home page
Task 22315301

Task 22315301

Name oifs_43r3_00e4_2019110100_123_993_12213946_0
Workunit 12213946
Created 24 Feb 2023, 13:16:25 UTC
Sent 24 Feb 2023, 13:20:06 UTC
Report deadline 25 Apr 2023, 13:20:06 UTC
Received 25 Feb 2023, 15:06:03 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1533319
Run time 23 hours 47 min 51 sec
CPU time 23 hours 32 min 46 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,413.16 MB
Peak swap size 4,984.13 MB
Peak disk usage 1,543.37 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
TEP 2530 H=2530:00 +CPU= 25.582
  11:41:07 STEP 2531 H=2531:00 +CPU= 23.823
  11:41:49 STEP 2532 H=2532:00 +CPU= 41.514
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002532
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002532
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002532
  11:42:10 STEP 2533 H=2533:00 +CPU= 20.462
  11:42:33 STEP 2534 H=2534:00 +CPU= 23.035
  11:43:09 STEP 2535 H=2535:00 +CPU= 35.425
  11:43:28 STEP 2536 H=2536:00 +CPU= 19.313
  11:43:51 STEP 2537 H=2537:00 +CPU= 22.715
  11:44:25 STEP 2538 H=2538:00 +CPU= 34.113
  11:44:49 STEP 2539 H=2539:00 +CPU= 23.968
  11:45:08 STEP 2540 H=2540:00 +CPU= 19.355
  11:45:44 STEP 2541 H=2541:00 +CPU= 35.712
  11:46:10 STEP 2542 H=2542:00 +CPU= 25.519
  11:46:39 STEP 2543 H=2543:00 +CPU= 21.374
  11:47:38 STEP 2544 H=2544:00 +CPU= 40.167
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002544
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_105.zip
Uploading the intermediate file: upload_file_105.zip
  11:47:56 STEP 2545 H=2545:00 +CPU= 17.661
  11:48:18 STEP 2546 H=2546:00 +CPU= 21.970
  11:48:50 STEP 2547 H=2547:00 +CPU= 31.540
  11:49:11 STEP 2548 H=2548:00 +CPU= 20.541
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215010000  !!!!!!!!!!!! 
 11:47:56 STEP 2545 H=2545:00 +CPU= 17.661
 NSTEP =  2546 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.992356776955E-01 1.764133908891E-01 6.506852936298E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.206349153952E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -2.12363897357062E-02 -1.93031090123818E+00  1.81392005027710E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.71297673924614E-02 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215011500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215013000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215014500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215020000  !!!!!!!!!!!! 
 11:48:18 STEP 2546 H=2546:00 +CPU= 21.970
  DATE= 2020    2   15
ISTEP=    2547: Reset RSOLINC= 1360.9440 RIP0= 1395.3549
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.248620232976000E-004
  9.609089255168670E-007  4.998719107099000E-007  9.565112577734999E-010
  2.022325970022666E-009  7.940283000000001E-011
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20191101
      152910  1.160825062034740E-004
SUECSO4: USE AVAILABLE FILES FOR DECADES:   2010 2020
 SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE:   2.334077972509880E-003
 NSTEP =  2547 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.885034335372E-01 1.797843825039E-01 6.508037619542E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.124080204300E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.40259026272358E-02 -1.79812783901466E+00  1.79007783003177E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.21718230181147E-02 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215021500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215023000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215024500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215030000  !!!!!!!!!!!! 
 11:48:50 STEP 2547 H=2547:00 +CPU= 31.540
 NSTEP =  2548 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.763230498429E-01 1.819041667890E-01 6.446668672012E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.074259363755E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -1.53794183473570E-03 -1.95638179872081E+00  1.98231981017368E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -2.09382797915972E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215031500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215033000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215034500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200215040000  !!!!!!!!!!!! 
 11:49:11 STEP 2548 H=2548:00 +CPU= 20.541
 NSTEP =  2549 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.036780348507E-01 1.814192043977E-01 6.433798028183E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.291803897661E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -8.38026522480319E-03 -2.13376031630314E+00  2.09200182334852E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60 -8.05719698285185E-03 -1.00000000000000E+00  1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 11:47:00 FULLPOS-B DYNFPOS   2544     1.144    1.144    1.144     33:40     36:29 0.12106242751998E-04       2GB       0MB
 11:47:01 0AAA00AAA STEPO     2544     1.455    1.455    1.517     33:42     36:30 0.12106242751998E-04       2GB       0MB
 11:47:38 0AAA00AAA STEPO     2545    34.990   34.990   36.414     34:17     37:06 0.12106242751998E-04       2GB       0MB
 11:47:56 0AAA00AAA STEPO     2546    17.661   17.661   17.670     34:34     37:24 0.12106242751998E-04       2GB       0MB
 11:48:19 0AAA00AAA STEPO     2547    21.971   21.971   23.118     34:56     37:47 0.12106242751998E-04       2GB       0MB
 11:48:50 0AAA00AAA STEPO     2548    31.539   31.539   31.609     35:28     38:19 0.12106242751998E-04       2GB       0MB
 11:49:11 0AAA00AAA STEPO     2549    20.541   20.541   20.628     35:48     38:40 0.12106242751998E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    2544",
 CTIME   = "01060000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98321.8522522047     ,
 GMASSI  =   98320.8442540264     
 /
------------------------------------------------
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00e4
(argv4) batchid: 993
(argv5) wuid: 12213946
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8b87a9707c1b63fc083a9b6344ac4bc0 to: /var/lib/boinc-client/slots/9/oifs_43r3_00e4_2019110100_123_993_12213946.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_00e4_2019110100_123_993_12213946.zip
ic_ancil_file: ic_ancil_12213946
ifsdata_file: ifsdata_12213946
climate_data_file: clim_data_12213946
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_cf579bc46664679dd047242dd1bf8173 to: /var/lib/boinc-client/slots/9/ic_ancil_12213946.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12213946.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9890438cd367e956c201c762e3002a52 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12213946.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12213946.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12213946.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12213946.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12213946.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12213946.xml
-- Model is restarting --
Adjusting last_iter, 2549, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12213946.xml
last_cpu_time: 73912
upload_file_number: 106
last_iter: 2545
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175
The child process has been launched with process id: 15451
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:15451:15451] [20230225:120725:1677323245:0.001] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:09:50 STEP 2544 H=2544:00 +CPU= 29.186
  12:10:07 STEP 2545 H=2545:00 +CPU= 16.322
  12:10:26 STEP 2546 H=2546:00 +CPU= 18.862
  12:11:01 STEP 2547 H=2547:00 +CPU= 26.912
  12:11:20 STEP 2548 H=2548:00 +CPU= 18.955
  12:11:40 STEP 2549 H=2549:00 +CPU= 19.277
  12:12:08 STEP 2550 H=2550:00 +CPU= 28.805
  12:12:27 STEP 2551 H=2551:00 +CPU= 18.897
  12:12:47 STEP 2552 H=2552:00 +CPU= 19.375
  12:13:14 STEP 2553 H=2553:00 +CPU= 27.077
  12:13:39 STEP 2554 H=2554:00 +CPU= 25.356
  12:14:04 STEP 2555 H=2555:00 +CPU= 24.592
  12:14:47 STEP 2556 H=2556:00 +CPU= 42.169
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002556
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002556
  12:15:06 STEP 2557 H=2557:00 +CPU= 19.670
  12:15:28 STEP 2558 H=2558:00 +CPU= 22.061
  12:15:57 STEP 2559 H=2559:00 +CPU= 28.468
  12:16:17 STEP 2560 H=2560:00 +CPU= 19.844
  12:16:43 STEP 2561 H=2561:00 +CPU= 25.667
  12:17:21 STEP 2562 H=2562:00 +CPU= 38.656
  12:17:39 STEP 2563 H=2563:00 +CPU= 17.395
  12:17:58 STEP 2564 H=2564:00 +CPU= 19.297
  12:18:33 STEP 2565 H=2565:00 +CPU= 34.633
  12:18:56 STEP 2566 H=2566:00 +CPU= 22.284
  12:19:20 STEP 2567 H=2567:00 +CPU= 24.488
  12:19:57 STEP 2568 H=2568:00 +CPU= 30.603
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002568
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_106.zip
Uploading the intermediate file: upload_file_106.zip
  12:20:14 STEP 2569 H=2569:00 +CPU= 16.541
  12:20:35 STEP 2570 H=2570:00 +CPU= 20.590
  12:21:05 STEP 2571 H=2571:00 +CPU= 30.376
  12:21:27 STEP 2572 H=2572:00 +CPU= 21.564
  12:21:53 STEP 2573 H=2573:00 +CPU= 26.076
  12:22:25 STEP 2574 H=2574:00 +CPU= 31.874
  12:22:45 STEP 2575 H=2575:00 +CPU= 20.465
  12:23:11 STEP 2576 H=2576:00 +CPU= 26.143
  12:23:50 STEP 2577 H=2577:00 +CPU= 38.607
  12:24:16 STEP 2578 H=2578:00 +CPU= 25.738
  12:24:35 STEP 2579 H=2579:00 +CPU= 19.554
  12:25:13 STEP 2580 H=2580:00 +CPU= 37.400
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002580
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002580
  12:25:35 STEP 2581 H=2581:00 +CPU= 22.234
  12:25:55 STEP 2582 H=2582:00 +CPU= 19.911
  12:26:34 STEP 2583 H=2583:00 +CPU= 38.117
  12:27:00 STEP 2584 H=2584:00 +CPU= 26.192
  12:27:26 STEP 2585 H=2585:00 +CPU= 25.744
  12:28:05 STEP 2586 H=2586:00 +CPU= 38.889
  12:28:23 STEP 2587 H=2587:00 +CPU= 17.818
  12:28:41 STEP 2588 H=2588:00 +CPU= 18.636
  12:29:18 STEP 2589 H=2589:00 +CPU= 36.061
  12:29:43 STEP 2590 H=2590:00 +CPU= 25.464
  12:30:08 STEP 2591 H=2591:00 +CPU= 25.290
  12:30:47 STEP 2592 H=2592:00 +CPU= 34.382
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002592
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_107.zip
Uploading the intermediate file: upload_file_107.zip
  12:31:14 STEP 2593 H=2593:00 +CPU= 22.506
  12:31:35 STEP 2594 H=2594:00 +CPU= 20.942
  12:32:08 STEP 2595 H=2595:00 +CPU= 32.744
  12:32:32 STEP 2596 H=2596:00 +CPU= 24.726
  12:32:51 STEP 2597 H=2597:00 +CPU= 18.692
  12:33:19 STEP 2598 H=2598:00 +CPU= 28.099
  12:33:36 STEP 2599 H=2599:00 +CPU= 16.947
  12:33:58 STEP 2600 H=2600:00 +CPU= 21.459
  12:34:34 STEP 2601 H=2601:00 +CPU= 36.716
  12:34:56 STEP 2602 H=2602:00 +CPU= 21.000
  12:35:20 STEP 2603 H=2603:00 +CPU= 24.759
  12:36:00 STEP 2604 H=2604:00 +CPU= 39.797
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002604
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002604
  12:36:20 STEP 2605 H=2605:00 +CPU= 19.929
  12:36:45 STEP 2606 H=2606:00 +CPU= 24.356
  12:37:23 STEP 2607 H=2607:00 +CPU= 38.419
  12:37:45 STEP 2608 H=2608:00 +CPU= 21.593
  12:38:10 STEP 2609 H=2609:00 +CPU= 25.086
  12:38:50 STEP 2610 H=2610:00 +CPU= 39.664
  12:39:09 STEP 2611 H=2611:00 +CPU= 19.077
  12:39:28 STEP 2612 H=2612:00 +CPU= 18.503
  12:40:06 STEP 2613 H=2613:00 +CPU= 38.297
  12:40:33 STEP 2614 H=2614:00 +CPU= 27.103
  12:40:52 STEP 2615 H=2615:00 +CPU= 18.909
  12:41:41 STEP 2616 H=2616:00 +CPU= 40.454
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002616
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_108.zip
Uploading the intermediate file: upload_file_108.zip
  12:41:58 STEP 2617 H=2617:00 +CPU= 17.841
  12:42:19 STEP 2618 H=2618:00 +CPU= 19.908
  12:42:53 STEP 2619 H=2619:00 +CPU= 34.601
  12:43:20 STEP 2620 H=2620:00 +CPU= 26.504
  12:43:45 STEP 2621 H=2621:00 +CPU= 25.239
  12:44:21 STEP 2622 H=2622:00 +CPU= 35.848
  12:44:43 STEP 2623 H=2623:00 +CPU= 21.927
  12:45:01 STEP 2624 H=2624:00 +CPU= 18.057
  12:45:30 STEP 2625 H=2625:00 +CPU= 28.401
  12:45:48 STEP 2626 H=2626:00 +CPU= 18.420
  12:46:09 STEP 2627 H=2627:00 +CPU= 21.058
  12:46:42 STEP 2628 H=2628:00 +CPU= 32.534
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002628
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002628
  12:47:08 STEP 2629 H=2629:00 +CPU= 26.389
  12:47:31 STEP 2630 H=2630:00 +CPU= 22.283
  12:48:09 STEP 2631 H=2631:00 +CPU= 38.094
  12:48:35 STEP 2632 H=2632:00 +CPU= 25.656
  12:49:00 STEP 2633 H=2633:00 +CPU= 25.489
  12:49:41 STEP 2634 H=2634:00 +CPU= 40.447
  12:50:03 STEP 2635 H=2635:00 +CPU= 22.368
  12:50:21 STEP 2636 H=2636:00 +CPU= 17.546
  12:50:50 STEP 2637 H=2637:00 +CPU= 29.037
  12:51:12 STEP 2638 H=2638:00 +CPU= 22.217
  12:51:35 STEP 2639 H=2639:00 +CPU= 22.616
  12:52:15 STEP 2640 H=2640:00 +CPU= 34.944
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002640
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_109.zip
Uploading the intermediate file: upload_file_109.zip
  12:52:37 STEP 2641 H=2641:00 +CPU= 21.813
  12:53:02 STEP 2642 H=2642:00 +CPU= 24.822
  12:53:35 STEP 2643 H=2643:00 +CPU= 33.370
  12:53:53 STEP 2644 H=2644:00 +CPU= 17.701
  12:54:14 STEP 2645 H=2645:00 +CPU= 20.973
  12:54:54 STEP 2646 H=2646:00 +CPU= 40.095
  12:55:19 STEP 2647 H=2647:00 +CPU= 24.693
  12:55:46 STEP 2648 H=2648:00 +CPU= 26.682
  12:56:25 STEP 2649 H=2649:00 +CPU= 39.400
  12:56:48 STEP 2650 H=2650:00 +CPU= 22.319
  12:57:05 STEP 2651 H=2651:00 +CPU= 17.154
  12:57:40 STEP 2652 H=2652:00 +CPU= 35.212
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002652
  12:58:01 STEP 2653 H=2653:00 +CPU= 20.618
  12:58:20 STEP 2654 H=2654:00 +CPU= 18.771
  12:58:53 STEP 2655 H=2655:00 +CPU= 33.243
  12:59:18 STEP 2656 H=2656:00 +CPU= 25.115
  12:59:43 STEP 2657 H=2657:00 +CPU= 24.671
  13:00:23 STEP 2658 H=2658:00 +CPU= 39.547
  13:00:48 STEP 2659 H=2659:00 +CPU= 24.943
  13:01:08 STEP 2660 H=2660:00 +CPU= 20.678
  13:01:47 STEP 2661 H=2661:00 +CPU= 38.477
  13:02:10 STEP 2662 H=2662:00 +CPU= 22.536
  13:02:36 STEP 2663 H=2663:00 +CPU= 26.406
  13:03:29 STEP 2664 H=2664:00 +CPU= 44.280
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002664
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_110.zip
Uploading the intermediate file: upload_file_110.zip
  13:03:46 STEP 2665 H=2665:00 +CPU= 17.296
  13:04:04 STEP 2666 H=2666:00 +CPU= 17.783
  13:04:39 STEP 2667 H=2667:00 +CPU= 34.728
  13:05:00 STEP 2668 H=2668:00 +CPU= 21.613
  13:05:22 STEP 2669 H=2669:00 +CPU= 21.751
  13:05:50 STEP 2670 H=2670:00 +CPU= 27.852
  13:06:13 STEP 2671 H=2671:00 +CPU= 23.318
  13:06:39 STEP 2672 H=2672:00 +CPU= 25.492
  13:07:18 STEP 2673 H=2673:00 +CPU= 38.721
  13:07:43 STEP 2674 H=2674:00 +CPU= 25.102
  13:08:08 STEP 2675 H=2675:00 +CPU= 24.883
  13:08:52 STEP 2676 H=2676:00 +CPU= 43.967
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002676
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002676
  13:09:18 STEP 2677 H=2677:00 +CPU= 25.689
  13:09:37 STEP 2678 H=2678:00 +CPU= 19.610
  13:10:09 STEP 2679 H=2679:00 +CPU= 31.135
  13:10:34 STEP 2680 H=2680:00 +CPU= 25.241
  13:10:59 STEP 2681 H=2681:00 +CPU= 24.691
  13:11:38 STEP 2682 H=2682:00 +CPU= 39.410
  13:12:00 STEP 2683 H=2683:00 +CPU= 21.368
  13:12:18 STEP 2684 H=2684:00 +CPU= 18.561
  13:12:56 STEP 2685 H=2685:00 +CPU= 37.442
  13:13:17 STEP 2686 H=2686:00 +CPU= 21.588
  13:13:43 STEP 2687 H=2687:00 +CPU= 25.974
  13:14:29 STEP 2688 H=2688:00 +CPU= 40.246
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002688
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_111.zip
Uploading the intermediate file: upload_file_111.zip
  13:14:54 STEP 2689 H=2689:00 +CPU= 25.333
  13:15:15 STEP 2690 H=2690:00 +CPU= 20.616
  13:15:54 STEP 2691 H=2691:00 +CPU= 38.972
  13:16:17 STEP 2692 H=2692:00 +CPU= 23.458
  13:16:43 STEP 2693 H=2693:00 +CPU= 25.445
  13:17:23 STEP 2694 H=2694:00 +CPU= 39.688
  13:17:48 STEP 2695 H=2695:00 +CPU= 25.193
  13:18:09 STEP 2696 H=2696:00 +CPU= 20.783
  13:18:47 STEP 2697 H=2697:00 +CPU= 37.972
  13:19:11 STEP 2698 H=2698:00 +CPU= 24.625
  13:19:31 STEP 2699 H=2699:00 +CPU= 19.697
  13:20:13 STEP 2700 H=2700:00 +CPU= 41.924
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002700
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002700
  13:20:32 STEP 2701 H=2701:00 +CPU= 18.586
  13:20:49 STEP 2702 H=2702:00 +CPU= 17.840
  13:21:18 STEP 2703 H=2703:00 +CPU= 28.560
  13:21:37 STEP 2704 H=2704:00 +CPU= 18.384
  13:21:55 STEP 2705 H=2705:00 +CPU= 18.344
  13:22:24 STEP 2706 H=2706:00 +CPU= 29.067
  13:22:48 STEP 2707 H=2707:00 +CPU= 24.060
  13:23:13 STEP 2708 H=2708:00 +CPU= 24.539
  13:23:45 STEP 2709 H=2709:00 +CPU= 31.474
  13:24:11 STEP 2710 H=2710:00 +CPU= 26.961
  13:24:38 STEP 2711 H=2711:00 +CPU= 26.529
  13:25:26 STEP 2712 H=2712:00 +CPU= 40.167
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002712
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_112.zip
Uploading the intermediate file: upload_file_112.zip
  13:25:53 STEP 2713 H=2713:00 +CPU= 24.209
  13:26:15 STEP 2714 H=2714:00 +CPU= 22.060
  13:26:54 STEP 2715 H=2715:00 +CPU= 39.019
  13:27:15 STEP 2716 H=2716:00 +CPU= 20.434
  13:27:32 STEP 2717 H=2717:00 +CPU= 17.521
  13:28:05 STEP 2718 H=2718:00 +CPU= 32.192
  13:28:23 STEP 2719 H=2719:00 +CPU= 18.438
  13:28:42 STEP 2720 H=2720:00 +CPU= 18.478
  13:29:21 STEP 2721 H=2721:00 +CPU= 39.726
  13:29:48 STEP 2722 H=2722:00 +CPU= 26.078
  13:30:11 STEP 2723 H=2723:00 +CPU= 23.671
  13:30:49 STEP 2724 H=2724:00 +CPU= 37.908
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002724
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002724
  13:31:14 STEP 2725 H=2725:00 +CPU= 25.136
  13:31:35 STEP 2726 H=2726:00 +CPU= 20.383
  13:32:03 STEP 2727 H=2727:00 +CPU= 28.089
  13:32:24 STEP 2728 H=2728:00 +CPU= 20.305
  13:32:46 STEP 2729 H=2729:00 +CPU= 22.317
  13:33:21 STEP 2730 H=2730:00 +CPU= 34.221
  13:33:38 STEP 2731 H=2731:00 +CPU= 17.597
  13:34:02 STEP 2732 H=2732:00 +CPU= 23.266
  13:34:37 STEP 2733 H=2733:00 +CPU= 35.704
  13:35:02 STEP 2734 H=2734:00 +CPU= 23.952
  13:35:27 STEP 2735 H=2735:00 +CPU= 25.592
  13:36:11 STEP 2736 H=2736:00 +CPU= 36.749
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002736
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_113.zip
Uploading the intermediate file: upload_file_113.zip
  13:36:36 STEP 2737 H=2737:00 +CPU= 24.969
  13:37:02 STEP 2738 H=2738:00 +CPU= 25.945
  13:37:37 STEP 2739 H=2739:00 +CPU= 34.772
  13:38:02 STEP 2740 H=2740:00 +CPU= 25.115
  13:38:28 STEP 2741 H=2741:00 +CPU= 25.910
  13:38:59 STEP 2742 H=2742:00 +CPU= 30.700
  13:39:17 STEP 2743 H=2743:00 +CPU= 17.898
  13:39:36 STEP 2744 H=2744:00 +CPU= 18.754
  13:40:09 STEP 2745 H=2745:00 +CPU= 32.647
  13:40:28 STEP 2746 H=2746:00 +CPU= 19.014
  13:40:51 STEP 2747 H=2747:00 +CPU= 23.414
  13:41:30 STEP 2748 H=2748:00 +CPU= 39.104
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002748
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002748
  13:41:56 STEP 2749 H=2749:00 +CPU= 25.798
  13:42:16 STEP 2750 H=2750:00 +CPU= 19.538
  13:42:54 STEP 2751 H=2751:00 +CPU= 38.554
  13:43:19 STEP 2752 H=2752:00 +CPU= 25.024
  13:43:44 STEP 2753 H=2753:00 +CPU= 24.927
  13:44:15 STEP 2754 H=2754:00 +CPU= 30.429
  13:44:32 STEP 2755 H=2755:00 +CPU= 16.898
  13:44:57 STEP 2756 H=2756:00 +CPU= 24.880
  13:45:30 STEP 2757 H=2757:00 +CPU= 33.636
  13:45:54 STEP 2758 H=2758:00 +CPU= 23.345
  13:46:15 STEP 2759 H=2759:00 +CPU= 21.649
  13:46:59 STEP 2760 H=2760:00 +CPU= 37.043
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002760
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_114.zip
Uploading the intermediate file: upload_file_114.zip
  13:47:24 STEP 2761 H=2761:00 +CPU= 24.950
  13:47:50 STEP 2762 H=2762:00 +CPU= 26.162
  13:48:24 STEP 2763 H=2763:00 +CPU= 33.606
  13:48:49 STEP 2764 H=2764:00 +CPU= 25.465
  13:49:10 STEP 2765 H=2765:00 +CPU= 21.227
  13:49:44 STEP 2766 H=2766:00 +CPU= 33.315
  13:50:08 STEP 2767 H=2767:00 +CPU= 24.353
  13:50:31 STEP 2768 H=2768:00 +CPU= 22.774
  13:51:11 STEP 2769 H=2769:00 +CPU= 40.329
  13:51:36 STEP 2770 H=2770:00 +CPU= 25.033
  13:52:02 STEP 2771 H=2771:00 +CPU= 25.515
  13:52:44 STEP 2772 H=2772:00 +CPU= 41.687
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002772
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002772
  13:53:03 STEP 2773 H=2773:00 +CPU= 19.586
  13:53:21 STEP 2774 H=2774:00 +CPU= 18.196
  13:54:00 STEP 2775 H=2775:00 +CPU= 38.362
  13:54:26 STEP 2776 H=2776:00 +CPU= 26.023
  13:54:52 STEP 2777 H=2777:00 +CPU= 25.605
  13:55:32 STEP 2778 H=2778:00 +CPU= 40.449
  13:55:58 STEP 2779 H=2779:00 +CPU= 25.242
  13:56:20 STEP 2780 H=2780:00 +CPU= 22.558
  13:56:55 STEP 2781 H=2781:00 +CPU= 34.665
  13:57:20 STEP 2782 H=2782:00 +CPU= 25.461
  13:57:44 STEP 2783 H=2783:00 +CPU= 23.341
  13:58:30 STEP 2784 H=2784:00 +CPU= 39.853
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002784
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_115.zip
Uploading the intermediate file: upload_file_115.zip
  13:58:53 STEP 2785 H=2785:00 +CPU= 22.827
Uploading trickle at timestep: 10022400
  13:59:12 STEP 2786 H=2786:00 +CPU= 18.409
  13:59:46 STEP 2787 H=2787:00 +CPU= 33.510
  14:00:06 STEP 2788 H=2788:00 +CPU= 20.406
  14:00:30 STEP 2789 H=2789:00 +CPU= 24.112
  14:00:59 STEP 2790 H=2790:00 +CPU= 28.755
  14:01:17 STEP 2791 H=2791:00 +CPU= 17.869
  14:01:39 STEP 2792 H=2792:00 +CPU= 22.313
  14:02:20 STEP 2793 H=2793:00 +CPU= 40.630
  14:02:44 STEP 2794 H=2794:00 +CPU= 24.293
  14:03:04 STEP 2795 H=2795:00 +CPU= 19.648
  14:03:46 STEP 2796 H=2796:00 +CPU= 42.118
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002796
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002796
  14:04:10 STEP 2797 H=2797:00 +CPU= 24.029
  14:04:36 STEP 2798 H=2798:00 +CPU= 25.856
  14:05:09 STEP 2799 H=2799:00 +CPU= 32.420
  14:05:32 STEP 2800 H=2800:00 +CPU= 23.170
  14:05:50 STEP 2801 H=2801:00 +CPU= 18.330
  14:06:30 STEP 2802 H=2802:00 +CPU= 39.103
  14:06:49 STEP 2803 H=2803:00 +CPU= 19.558
  14:07:09 STEP 2804 H=2804:00 +CPU= 19.766
  14:07:41 STEP 2805 H=2805:00 +CPU= 31.410
  14:08:05 STEP 2806 H=2806:00 +CPU= 24.260
  14:08:28 STEP 2807 H=2807:00 +CPU= 23.124
  14:09:20 STEP 2808 H=2808:00 +CPU= 45.406
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002808
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_116.zip
Uploading the intermediate file: upload_file_116.zip
  14:09:45 STEP 2809 H=2809:00 +CPU= 25.496
  14:10:12 STEP 2810 H=2810:00 +CPU= 26.523
  14:10:43 STEP 2811 H=2811:00 +CPU= 30.994
  14:11:05 STEP 2812 H=2812:00 +CPU= 21.625
  14:11:30 STEP 2813 H=2813:00 +CPU= 24.929
  14:12:07 STEP 2814 H=2814:00 +CPU= 36.756
  14:12:27 STEP 2815 H=2815:00 +CPU= 20.297
  14:12:50 STEP 2816 H=2816:00 +CPU= 22.757
  14:13:22 STEP 2817 H=2817:00 +CPU= 31.824
  14:13:45 STEP 2818 H=2818:00 +CPU= 23.031
  14:14:10 STEP 2819 H=2819:00 +CPU= 25.358
  14:14:43 STEP 2820 H=2820:00 +CPU= 32.596
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002820
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002820
  14:15:01 STEP 2821 H=2821:00 +CPU= 17.803
  14:15:23 STEP 2822 H=2822:00 +CPU= 22.642
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           14  MAX ETADOT VERTICAL VEL.=   7.949833691748105E-008
  LON  =    47.5000000000000       degrees
  LAT  =    40.9343383242788       degrees
  14:16:00 STEP 2823 H=2823:00 +CPU= 36.246
  14:16:21 STEP 2824 H=2824:00 +CPU= 21.436
  14:16:46 STEP 2825 H=2825:00 +CPU= 24.307
  14:17:15 STEP 2826 H=2826:00 +CPU= 29.166
  14:17:33 STEP 2827 H=2827:00 +CPU= 17.424
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           19  MAX ETADOT VERTICAL VEL.=   1.110179882667715E-007
  LON  =    46.4062500000000       degrees
  LAT  =    43.1773092288349       degrees
  14:17:51 STEP 2828 H=2828:00 +CPU= 17.973
  14:18:17 STEP 2829 H=2829:00 +CPU= 26.629
  14:18:43 STEP 2830 H=2830:00 +CPU= 25.969
  14:19:04 STEP 2831 H=2831:00 +CPU= 20.567
  14:19:56 STEP 2832 H=2832:00 +CPU= 38.153
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002832
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_117.zip
Uploading the intermediate file: upload_file_117.zip
  14:20:19 STEP 2833 H=2833:00 +CPU= 23.172
  14:20:39 STEP 2834 H=2834:00 +CPU= 20.517
  14:21:17 STEP 2835 H=2835:00 +CPU= 35.641
  14:21:42 STEP 2836 H=2836:00 +CPU= 25.225
  14:22:03 STEP 2837 H=2837:00 +CPU= 20.978
  14:22:41 STEP 2838 H=2838:00 +CPU= 38.355
  14:23:02 STEP 2839 H=2839:00 +CPU= 20.272
  14:23:24 STEP 2840 H=2840:00 +CPU= 21.795
  14:24:00 STEP 2841 H=2841:00 +CPU= 36.730
  14:24:25 STEP 2842 H=2842:00 +CPU= 24.116
  14:24:42 STEP 2843 H=2843:00 +CPU= 16.992
  14:25:12 STEP 2844 H=2844:00 +CPU= 30.472
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002844
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002844
  14:25:35 STEP 2845 H=2845:00 +CPU= 22.450
  14:25:54 STEP 2846 H=2846:00 +CPU= 18.794
  14:26:25 STEP 2847 H=2847:00 +CPU= 31.068
  14:26:47 STEP 2848 H=2848:00 +CPU= 21.814
  14:27:10 STEP 2849 H=2849:00 +CPU= 23.755
  14:27:45 STEP 2850 H=2850:00 +CPU= 34.482
  14:28:02 STEP 2851 H=2851:00 +CPU= 17.090
  14:28:19 STEP 2852 H=2852:00 +CPU= 17.161
  14:28:51 STEP 2853 H=2853:00 +CPU= 30.976
  14:29:17 STEP 2854 H=2854:00 +CPU= 26.189
  14:29:42 STEP 2855 H=2855:00 +CPU= 25.216
  14:30:35 STEP 2856 H=2856:00 +CPU= 39.999
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002856
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_118.zip
Uploading the intermediate file: upload_file_118.zip
  14:30:57 STEP 2857 H=2857:00 +CPU= 21.814
  14:31:15 STEP 2858 H=2858:00 +CPU= 18.041
  14:31:42 STEP 2859 H=2859:00 +CPU= 26.691
  14:32:00 STEP 2860 H=2860:00 +CPU= 18.299
  14:32:20 STEP 2861 H=2861:00 +CPU= 19.215
  14:33:00 STEP 2862 H=2862:00 +CPU= 40.348
  14:33:18 STEP 2863 H=2863:00 +CPU= 17.927
  14:33:39 STEP 2864 H=2864:00 +CPU= 21.250
  14:34:12 STEP 2865 H=2865:00 +CPU= 32.495
  14:34:39 STEP 2866 H=2866:00 +CPU= 26.622
  14:34:57 STEP 2867 H=2867:00 +CPU= 18.755
  14:35:31 STEP 2868 H=2868:00 +CPU= 33.230
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002868
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002868
  14:35:57 STEP 2869 H=2869:00 +CPU= 25.776
  14:36:23 STEP 2870 H=2870:00 +CPU= 26.163
  14:36:52 STEP 2871 H=2871:00 +CPU= 28.720
  14:37:14 STEP 2872 H=2872:00 +CPU= 22.754
  14:37:40 STEP 2873 H=2873:00 +CPU= 25.441
  14:38:19 STEP 2874 H=2874:00 +CPU= 39.477
  14:38:44 STEP 2875 H=2875:00 +CPU= 25.005
  14:39:07 STEP 2876 H=2876:00 +CPU= 22.734
  14:39:44 STEP 2877 H=2877:00 +CPU= 36.877
  14:40:04 STEP 2878 H=2878:00 +CPU= 19.466
  14:40:23 STEP 2879 H=2879:00 +CPU= 19.402
  14:41:13 STEP 2880 H=2880:00 +CPU= 41.217
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002880
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_119.zip
Uploading the intermediate file: upload_file_119.zip
  14:41:41 STEP 2881 H=2881:00 +CPU= 24.172
  14:42:04 STEP 2882 H=2882:00 +CPU= 23.488
  14:42:41 STEP 2883 H=2883:00 +CPU= 36.390
  14:43:07 STEP 2884 H=2884:00 +CPU= 25.761
  14:43:31 STEP 2885 H=2885:00 +CPU= 24.328
  14:44:03 STEP 2886 H=2886:00 +CPU= 31.774
  14:44:22 STEP 2887 H=2887:00 +CPU= 18.839
  14:44:45 STEP 2888 H=2888:00 +CPU= 22.789
  14:45:12 STEP 2889 H=2889:00 +CPU= 26.861
  14:45:35 STEP 2890 H=2890:00 +CPU= 23.029
  14:45:59 STEP 2891 H=2891:00 +CPU= 24.230
  14:46:37 STEP 2892 H=2892:00 +CPU= 37.490
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002892
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002892
  14:47:02 STEP 2893 H=2893:00 +CPU= 24.924
  14:47:21 STEP 2894 H=2894:00 +CPU= 19.123
  14:47:53 STEP 2895 H=2895:00 +CPU= 32.365
  14:48:19 STEP 2896 H=2896:00 +CPU= 25.597
  14:48:43 STEP 2897 H=2897:00 +CPU= 24.053
  14:49:23 STEP 2898 H=2898:00 +CPU= 39.605
  14:49:47 STEP 2899 H=2899:00 +CPU= 24.414
  14:50:07 STEP 2900 H=2900:00 +CPU= 20.102
  14:50:36 STEP 2901 H=2901:00 +CPU= 27.843
  14:50:57 STEP 2902 H=2902:00 +CPU= 21.134
  14:51:16 STEP 2903 H=2903:00 +CPU= 19.188
  14:52:00 STEP 2904 H=2904:00 +CPU= 38.921
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002904
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_120.zip
Uploading the intermediate file: upload_file_120.zip
  14:52:20 STEP 2905 H=2905:00 +CPU= 19.319
  14:52:40 STEP 2906 H=2906:00 +CPU= 20.231
  14:53:13 STEP 2907 H=2907:00 +CPU= 32.153
  14:53:31 STEP 2908 H=2908:00 +CPU= 18.860
  14:53:50 STEP 2909 H=2909:00 +CPU= 18.032
  14:54:24 STEP 2910 H=2910:00 +CPU= 33.827
  14:54:49 STEP 2911 H=2911:00 +CPU= 25.117
  14:55:16 STEP 2912 H=2912:00 +CPU= 26.762
  14:55:54 STEP 2913 H=2913:00 +CPU= 37.930
  14:56:13 STEP 2914 H=2914:00 +CPU= 19.764
  14:56:35 STEP 2915 H=2915:00 +CPU= 21.323
  14:57:17 STEP 2916 H=2916:00 +CPU= 42.029
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002916
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002916
  14:57:42 STEP 2917 H=2917:00 +CPU= 25.551
  14:58:08 STEP 2918 H=2918:00 +CPU= 25.302
  14:58:35 STEP 2919 H=2919:00 +CPU= 26.923
  14:58:55 STEP 2920 H=2920:00 +CPU= 20.580
  14:59:21 STEP 2921 H=2921:00 +CPU= 25.866
  14:59:58 STEP 2922 H=2922:00 +CPU= 36.417
  15:00:22 STEP 2923 H=2923:00 +CPU= 23.923
  15:00:42 STEP 2924 H=2924:00 +CPU= 20.095
  15:01:14 STEP 2925 H=2925:00 +CPU= 32.081
  15:01:35 STEP 2926 H=2926:00 +CPU= 20.263
  15:02:02 STEP 2927 H=2927:00 +CPU= 27.024
  15:02:52 STEP 2928 H=2928:00 +CPU= 44.192
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002928
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_121.zip
Uploading the intermediate file: upload_file_121.zip
  15:03:12 STEP 2929 H=2929:00 +CPU= 19.075
  15:03:32 STEP 2930 H=2930:00 +CPU= 20.519
  15:04:02 STEP 2931 H=2931:00 +CPU= 30.094
  15:04:20 STEP 2932 H=2932:00 +CPU= 17.246
  15:04:37 STEP 2933 H=2933:00 +CPU= 16.841
  15:05:09 STEP 2934 H=2934:00 +CPU= 32.133
  15:05:33 STEP 2935 H=2935:00 +CPU= 24.417
  15:05:54 STEP 2936 H=2936:00 +CPU= 20.942
  15:06:31 STEP 2937 H=2937:00 +CPU= 36.457
  15:06:48 STEP 2938 H=2938:00 +CPU= 17.839
  15:07:06 STEP 2939 H=2939:00 +CPU= 17.676
  15:07:40 STEP 2940 H=2940:00 +CPU= 33.455
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002940
  15:08:05 STEP 2941 H=2941:00 +CPU= 24.826
  15:08:30 STEP 2942 H=2942:00 +CPU= 25.619
  15:09:08 STEP 2943 H=2943:00 +CPU= 37.983
  15:09:26 STEP 2944 H=2944:00 +CPU= 17.665
  15:09:46 STEP 2945 H=2945:00 +CPU= 19.650
  15:10:16 STEP 2946 H=2946:00 +CPU= 30.326
  15:10:38 STEP 2947 H=2947:00 +CPU= 21.657
  15:11:03 STEP 2948 H=2948:00 +CPU= 24.686
  15:11:36 STEP 2949 H=2949:00 +CPU= 33.611
  15:12:02 STEP 2950 H=2950:00 +CPU= 25.726
  15:12:28 STEP 2951 H=2951:00 +CPU= 25.577
  15:13:02 STEP 2952 H=2952:00 +CPU= 33.895
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.05448946882639     
  MAXGPFV : MAX. VALUE =   0.739223484291902     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    342251210 BYTES IN      8 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0113( 2.7%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 15:13:02 STEP 2952 H=2952:00 +CPU= 33.895
 END CNT3
 NSTEP =  2952 CNT0   000000000  29.303  29.303  29.459   0.995               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         6341/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    156    0       35       786125     6341 ************   142.9    81.8    61.1    82.2    77.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.26      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  443    2193.8    2193.8      8.73      0.00
   9 SPCM     - SPECTRAL COMP.                     408     372.8     372.8      1.37      0.00
  10 SCAN2M - PHYSICS                              409   11382.1   11382.1     41.80      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 34    4398.7    4398.7      1.34      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 18      20.6      20.6      0.00      0.00
  13 SCAN2M - RADIATION CALC.                      137   12111.5   12111.5     14.90      0.00
  17 GRIDFPOS IN CNT4                               34      44.6      44.6      0.01      0.00
  27 RESTART FILES - WRITING                        17    8797.0    8797.0      1.34      0.00
  28 RESTART FILES - READING                         1   82584.0   82584.0      0.74      0.00
  29 SU4FPOS IN CNT4                                34       1.6       1.6      0.00      0.00
  30 DYNFPOS IN CNT4                                34    2307.3    2307.3      0.70      0.00
  31 POSDDH IN STEPO                                 2     976.3     976.3      0.02      0.00
  37 CPGLAG   - SL COMPUTATIONS                    409    3596.5    3596.5     13.21      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL           409    3006.1    3006.1     11.04      0.00
  39 SU0YOMB                                         1     190.8     190.8      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    409      22.4      22.4      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              408     251.1     251.1      0.92      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              408      37.4      37.4      0.14      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 18       1.3       1.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1400     156.2     156.2      1.96      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       850     139.0     139.0      1.06      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        850      54.8      54.8      0.42      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1400      53.6      53.6      0.67      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      12.6      12.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1400      17.2      17.2      0.22      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            850      15.4      15.4      0.12      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1400      29.1      29.1      0.37      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            850      13.3      13.3      0.10      0.00
 400 GSTATS                                      82360       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 78820       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    11137.2 CPU TIME   10854.3 VECTOR TIME    10854.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      961.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhnw4+002952
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/slots/9/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/9/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMGGhnw4+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12213946/ICMSHhnw4+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_00e4_2019110100_123_993_12213946_0_r1842034175_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
15:16:24 (15046): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:53:44 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 10,623,600 84,766 0.0080
09 May 2023 16:53:38 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 10,022,400 80,373 0.0080
09 May 2023 16:53:22 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 7,776,000 61,637 0.0079
09 May 2023 16:53:13 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 6,912,000 54,585 0.0079
09 May 2023 16:53:06 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 6,048,000 47,683 0.0079
09 May 2023 16:53:00 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 5,184,000 40,648 0.0078
09 May 2023 16:52:54 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 4,320,000 33,744 0.0078
09 May 2023 16:52:49 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 3,456,000 26,869 0.0078
09 May 2023 16:52:45 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 2,592,000 19,873 0.0077
09 May 2023 16:52:42 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 1,728,000 13,225 0.0077
09 May 2023 16:52:39 1533319 22315301 oifs_43r3_00e4_2019110100_123_993_12213946_0 864,000 6,673 0.0077


©2024 climateprediction.net