climateprediction.net home page
Task 22443355

Task 22443355

Name oifs_43r3_bl_a141_2016092300_20_1018_12289768_0
Workunit 12289768
Created 12 Jun 2024, 22:03:00 UTC
Sent 13 Jun 2024, 6:00:27 UTC
Report deadline 12 Aug 2024, 6:00:27 UTC
Received 20 Jun 2024, 10:28:19 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1496581
Run time 18 hours 48 min 3 sec
CPU time 18 hours 48 min 3 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 4.08 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,022.55 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,284.29 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
STEP 1692 H= 423:00 +CPU= 35.915
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001692
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001692
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:36:01 STEP 1693 H= 423:15 +CPU= 30.089
  00:36:32 STEP 1694 H= 423:30 +CPU= 28.951
  00:37:01 STEP 1695 H= 423:45 +CPU= 28.759
  00:37:30 STEP 1696 H= 424:00 +CPU= 27.985
  00:37:57 STEP 1697 H= 424:15 +CPU= 26.864
  00:38:25 STEP 1698 H= 424:30 +CPU= 27.216
  00:38:52 STEP 1699 H= 424:45 +CPU= 27.517
  00:39:21 STEP 1700 H= 425:00 +CPU= 28.250
  00:39:48 STEP 1701 H= 425:15 +CPU= 27.029
  00:40:19 STEP 1702 H= 425:30 +CPU= 29.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:41:10 STEP 1703 H= 425:45 +CPU= 28.295
  00:41:45 STEP 1704 H= 426:00 +CPU= 34.436
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001704
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001704
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001704
  00:42:14 STEP 1705 H= 426:15 +CPU= 28.453
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:43:04 STEP 1706 H= 426:30 +CPU= 27.902
  00:43:31 STEP 1707 H= 426:45 +CPU= 26.693
  00:44:02 STEP 1708 H= 427:00 +CPU= 29.848
  00:44:30 STEP 1709 H= 427:15 +CPU= 27.929
  00:44:59 STEP 1710 H= 427:30 +CPU= 28.246
  00:45:30 STEP 1711 H= 427:45 +CPU= 29.411
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005241:1718837561:85816.242] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3958MB (heap), 4797MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005241:1718837561:85816.269] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005241:1718837561:85816.269] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005241:1718837561:85816.269] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005241:1718837561:85816.269] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005241:1718837561:85816.495] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 4797 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.513] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.547] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005242:1718837562:85816.853] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005243:1718837563:85817.853] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:svarten.tribun:1:1:26455:26455] [20240620:005243:1718837563:85817.854] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000082EA3F  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000082D844  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008362A6  dr_hook_util_             126  dr_hook_util.F90
oifs_43r3_model.e  0000000000402B81  yomhook_mp_dr_hoo          59  yomhook.F90
oifs_43r3_model.e  00000000010AC70A  surfbc_layer_              76  surfbc_layer.F90
oifs_43r3_model.e  0000000000FA3241  callpar_                  621  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a141
(argv4) batchid: 1018
(argv5) wuid: 12289768
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /mnt/WDred/boinc/slots/0
Project directory is: /mnt/WDred/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768
..mkdir for temp folder for results failed
Copying: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_499f7785c7772e9c07e9737ddcc47260 to: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_a141_2016092300_20_1018_12289768.zip
Unzipping the namelist zip file: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_a141_2016092300_20_1018_12289768.zip
ic_ancil_file: ic_ancil_12289768
ifsdata_file: ifsdata_12289768
climate_data_file: clim_data_12289768
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /mnt/WDred/boinc/slots/0/ic_ancil_12289768.zip
Unzipping the IC ancils zip file: /mnt/WDred/boinc/slots/0/ic_ancil_12289768.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6aa1bfdd4749a7311f5243e700574d6 to: /mnt/WDred/boinc/slots/0/ifsdata/ifsdata_12289768.zip
Unzipping the ifsdata_zip file: /mnt/WDred/boinc/slots/0/ifsdata/ifsdata_12289768.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/WDred/boinc/slots/0/159l_2/clim_data_12289768.zip
Unzipping the climate data zip file: /mnt/WDred/boinc/slots/0/159l_2/clim_data_12289768.zip
Checking for progress XML file: /mnt/WDred/boinc/slots/0/progress_file_12289768.xml
Opened progress file ok : /mnt/WDred/boinc/slots/0/progress_file_12289768.xml
-- Model is restarting --
Adjusting last_iter, 1712, to previous model restart step.
Creating progress file: /mnt/WDred/boinc/slots/0/progress_file_12289768.xml
last_cpu_time: 59199
upload_file_number: 17
last_iter: 1681
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a141_2016092300_20_1018_12289768_0_r1702435919
The child process has been launched with process id: 204158
Executing the command: /mnt/WDred/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:204158:204158] [20240620:013259:1718839979:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:33:44 STEP 1680 H= 420:00 +CPU= 28.087
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a141
(argv4) batchid: 1018
(argv5) wuid: 12289768
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /mnt/WDred/boinc/slots/0
Project directory is: /mnt/WDred/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768
..mkdir for temp folder for results failed
Copying: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_499f7785c7772e9c07e9737ddcc47260 to: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_a141_2016092300_20_1018_12289768.zip
Unzipping the namelist zip file: /mnt/WDred/boinc/slots/0/oifs_43r3_bl_a141_2016092300_20_1018_12289768.zip
ic_ancil_file: ic_ancil_12289768
ifsdata_file: ifsdata_12289768
climate_data_file: clim_data_12289768
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /mnt/WDred/boinc/slots/0/ic_ancil_12289768.zip
Unzipping the IC ancils zip file: /mnt/WDred/boinc/slots/0/ic_ancil_12289768.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f6aa1bfdd4749a7311f5243e700574d6 to: /mnt/WDred/boinc/slots/0/ifsdata/ifsdata_12289768.zip
Unzipping the ifsdata_zip file: /mnt/WDred/boinc/slots/0/ifsdata/ifsdata_12289768.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/WDred/boinc/slots/0/159l_2/clim_data_12289768.zip
Unzipping the climate data zip file: /mnt/WDred/boinc/slots/0/159l_2/clim_data_12289768.zip
Checking for progress XML file: /mnt/WDred/boinc/slots/0/progress_file_12289768.xml
Opened progress file ok : /mnt/WDred/boinc/slots/0/progress_file_12289768.xml
-- Model is restarting --
Adjusting last_iter, 1681, to previous model restart step.
Creating progress file: /mnt/WDred/boinc/slots/0/progress_file_12289768.xml
last_cpu_time: 59235
upload_file_number: 17
last_iter: 1681
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a141_2016092300_20_1018_12289768_0_r1702435919
The child process has been launched with process id: 16696
Executing the command: /mnt/WDred/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:svarten.tribun:1:1:16696:16696] [20240620:080517:1718863517:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:06:13 STEP 1680 H= 420:00 +CPU= 38.452
  08:06:51 STEP 1681 H= 420:15 +CPU= 36.889
  08:07:29 STEP 1682 H= 420:30 +CPU= 36.813
  08:08:07 STEP 1683 H= 420:45 +CPU= 36.729
  08:08:45 STEP 1684 H= 421:00 +CPU= 37.503
  08:09:23 STEP 1685 H= 421:15 +CPU= 36.841
  08:10:01 STEP 1686 H= 421:30 +CPU= 36.806
  08:10:40 STEP 1687 H= 421:45 +CPU= 36.948
  08:11:19 STEP 1688 H= 422:00 +CPU= 37.629
  08:11:56 STEP 1689 H= 422:15 +CPU= 36.786
  08:12:34 STEP 1690 H= 422:30 +CPU= 36.897
  08:13:12 STEP 1691 H= 422:45 +CPU= 36.772
  08:13:59 STEP 1692 H= 423:00 +CPU= 46.512
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001692
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001692
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001692
  08:14:37 STEP 1693 H= 423:15 +CPU= 36.944
  08:15:16 STEP 1694 H= 423:30 +CPU= 36.965
  08:15:54 STEP 1695 H= 423:45 +CPU= 36.908
  08:16:33 STEP 1696 H= 424:00 +CPU= 37.650
  08:17:10 STEP 1697 H= 424:15 +CPU= 36.950
  08:17:48 STEP 1698 H= 424:30 +CPU= 36.919
  08:18:26 STEP 1699 H= 424:45 +CPU= 36.992
  08:19:05 STEP 1700 H= 425:00 +CPU= 37.651
  08:19:43 STEP 1701 H= 425:15 +CPU= 36.936
  08:20:21 STEP 1702 H= 425:30 +CPU= 37.016
  08:20:59 STEP 1703 H= 425:45 +CPU= 36.905
  08:21:47 STEP 1704 H= 426:00 +CPU= 46.472
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001704
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001704
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001704
  08:22:24 STEP 1705 H= 426:15 +CPU= 36.923
  08:23:02 STEP 1706 H= 426:30 +CPU= 36.939
  08:23:40 STEP 1707 H= 426:45 +CPU= 37.011
  08:24:19 STEP 1708 H= 427:00 +CPU= 37.735
  08:24:57 STEP 1709 H= 427:15 +CPU= 37.008
  08:25:35 STEP 1710 H= 427:30 +CPU= 37.020
  08:26:13 STEP 1711 H= 427:45 +CPU= 36.916
  08:26:52 STEP 1712 H= 428:00 +CPU= 37.693
  08:27:30 STEP 1713 H= 428:15 +CPU= 36.877
  08:28:08 STEP 1714 H= 428:30 +CPU= 36.967
  08:28:45 STEP 1715 H= 428:45 +CPU= 36.949
  08:29:33 STEP 1716 H= 429:00 +CPU= 46.437
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001716
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001716
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001716
  08:30:12 STEP 1717 H= 429:15 +CPU= 37.080
  08:30:50 STEP 1718 H= 429:30 +CPU= 36.947
  08:31:28 STEP 1719 H= 429:45 +CPU= 36.985
  08:32:07 STEP 1720 H= 430:00 +CPU= 37.737
  08:32:45 STEP 1721 H= 430:15 +CPU= 37.030
  08:33:22 STEP 1722 H= 430:30 +CPU= 37.037
  08:34:00 STEP 1723 H= 430:45 +CPU= 36.952
  08:34:39 STEP 1724 H= 431:00 +CPU= 37.777
  08:35:24 STEP 1725 H= 431:15 +CPU= 37.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:36:27 STEP 1726 H= 431:30 +CPU= 37.047
  08:37:05 STEP 1727 H= 431:45 +CPU= 37.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:38:59 STEP 1728 H= 432:00 +CPU= 47.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001728
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001728
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001728
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001632
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001632
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001632
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001644
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001644
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001644
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001656
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001656
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001656
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001668
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001668
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001668
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001680
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001680
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001680
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001692
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001692
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001692
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001704
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001704
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001704
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001716
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001716
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001716
Zipping up the intermediate file: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_a141_2016092300_20_1018_12289768_0_r1702435919_17.zip
Uploading the intermediate file: upload_file_17.zip
  08:39:55 STEP 1729 H= 432:15 +CPU= 32.171
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:41:32 STEP 1730 H= 432:30 +CPU= 29.610
  08:42:01 STEP 1731 H= 432:45 +CPU= 28.312
  08:42:31 STEP 1732 H= 433:00 +CPU= 28.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:00:15 STEP 1733 H= 433:15 +CPU= 33.105
  10:00:55 STEP 1734 H= 433:30 +CPU= 37.469
  10:01:35 STEP 1735 H= 433:45 +CPU= 37.553
  10:02:17 STEP 1736 H= 434:00 +CPU= 38.305
  10:02:55 STEP 1737 H= 434:15 +CPU= 37.098
  10:03:33 STEP 1738 H= 434:30 +CPU= 37.082
  10:04:12 STEP 1739 H= 434:45 +CPU= 37.086
  10:05:01 STEP 1740 H= 435:00 +CPU= 46.649
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001740
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001740
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001740
  10:05:41 STEP 1741 H= 435:15 +CPU= 37.099
  10:06:19 STEP 1742 H= 435:30 +CPU= 37.085
  10:06:58 STEP 1743 H= 435:45 +CPU= 37.072
  10:07:37 STEP 1744 H= 436:00 +CPU= 37.837
  10:08:16 STEP 1745 H= 436:15 +CPU= 37.111
  10:08:54 STEP 1746 H= 436:30 +CPU= 36.989
  10:09:33 STEP 1747 H= 436:45 +CPU= 37.117
  10:10:13 STEP 1748 H= 437:00 +CPU= 37.895
  10:10:51 STEP 1749 H= 437:15 +CPU= 37.081
  10:11:30 STEP 1750 H= 437:30 +CPU= 37.103
  10:12:08 STEP 1751 H= 437:45 +CPU= 37.068
  10:12:56 STEP 1752 H= 438:00 +CPU= 46.680
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001752
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001752
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001752
  10:13:35 STEP 1753 H= 438:15 +CPU= 37.086
  10:14:14 STEP 1754 H= 438:30 +CPU= 37.033
  10:14:52 STEP 1755 H= 438:45 +CPU= 37.033
  10:15:32 STEP 1756 H= 439:00 +CPU= 37.947
  10:16:11 STEP 1757 H= 439:15 +CPU= 37.083
  10:16:49 STEP 1758 H= 439:30 +CPU= 37.014
  10:17:27 STEP 1759 H= 439:45 +CPU= 37.034
  10:18:07 STEP 1760 H= 440:00 +CPU= 37.830
  10:18:45 STEP 1761 H= 440:15 +CPU= 37.055
  10:19:23 STEP 1762 H= 440:30 +CPU= 37.070
  10:20:02 STEP 1763 H= 440:45 +CPU= 37.076
  10:20:49 STEP 1764 H= 441:00 +CPU= 43.989
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001764
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001764
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001764
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:22:47 STEP 1765 H= 441:15 +CPU= 30.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:26:13 STEP 1766 H= 441:30 +CPU= 30.776
  10:26:43 STEP 1767 H= 441:45 +CPU= 29.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:27:37 STEP 1768 H= 442:00 +CPU= 30.560
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:28:31 STEP 1769 H= 442:15 +CPU= 30.978
  10:29:01 STEP 1770 H= 442:30 +CPU= 29.572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:29:57 STEP 1771 H= 442:45 +CPU= 31.581
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:30:51 STEP 1772 H= 443:00 +CPU= 29.321
  10:31:19 STEP 1773 H= 443:15 +CPU= 27.902
  10:31:49 STEP 1774 H= 443:30 +CPU= 29.342
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:32:42 STEP 1775 H= 443:45 +CPU= 29.855
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:34:06 STEP 1776 H= 444:00 +CPU= 40.274
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001776
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001776
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001776
  10:34:36 STEP 1777 H= 444:15 +CPU= 28.747
  10:35:08 STEP 1778 H= 444:30 +CPU= 29.714
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:37:23 STEP 1779 H= 444:45 +CPU= 30.433
  10:37:55 STEP 1780 H= 445:00 +CPU= 30.783
  10:38:25 STEP 1781 H= 445:15 +CPU= 28.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:39:42 STEP 1782 H= 445:30 +CPU= 31.995
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:40:36 STEP 1783 H= 445:45 +CPU= 30.138
  10:41:09 STEP 1784 H= 446:00 +CPU= 30.765
  10:41:39 STEP 1785 H= 446:15 +CPU= 28.794
  10:42:08 STEP 1786 H= 446:30 +CPU= 28.700
  10:42:38 STEP 1787 H= 446:45 +CPU= 29.149
  10:43:13 STEP 1788 H= 447:00 +CPU= 34.753
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001788
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001788
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001788
  10:43:42 STEP 1789 H= 447:15 +CPU= 27.779
  10:44:10 STEP 1790 H= 447:30 +CPU= 27.762
  10:44:38 STEP 1791 H= 447:45 +CPU= 27.941
  10:45:08 STEP 1792 H= 448:00 +CPU= 29.026
  10:45:36 STEP 1793 H= 448:15 +CPU= 27.837
  10:46:09 STEP 1794 H= 448:30 +CPU= 32.124
  10:46:48 STEP 1795 H= 448:45 +CPU= 37.338
  10:47:28 STEP 1796 H= 449:00 +CPU= 38.091
  10:47:57 STEP 1797 H= 449:15 +CPU= 28.429
  10:48:25 STEP 1798 H= 449:30 +CPU= 27.727
  10:48:55 STEP 1799 H= 449:45 +CPU= 28.579
  10:49:34 STEP 1800 H= 450:00 +CPU= 37.250
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001800
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001800
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001800
  10:50:05 STEP 1801 H= 450:15 +CPU= 29.661
  10:50:35 STEP 1802 H= 450:30 +CPU= 28.668
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:51:29 STEP 1803 H= 450:45 +CPU= 30.112
  10:52:01 STEP 1804 H= 451:00 +CPU= 30.593
  10:52:29 STEP 1805 H= 451:15 +CPU= 27.782
  10:52:59 STEP 1806 H= 451:30 +CPU= 28.984
  10:53:27 STEP 1807 H= 451:45 +CPU= 27.968
  10:53:57 STEP 1808 H= 452:00 +CPU= 28.901
  10:54:25 STEP 1809 H= 452:15 +CPU= 28.018
  10:54:54 STEP 1810 H= 452:30 +CPU= 28.270
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:55:46 STEP 1811 H= 452:45 +CPU= 29.656
  10:56:23 STEP 1812 H= 453:00 +CPU= 36.347
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001812
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001812
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001812
  10:56:53 STEP 1813 H= 453:15 +CPU= 29.367
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:57:46 STEP 1814 H= 453:30 +CPU= 29.898
  10:58:14 STEP 1815 H= 453:45 +CPU= 28.038
  10:58:43 STEP 1816 H= 454:00 +CPU= 28.231
  10:59:12 STEP 1817 H= 454:15 +CPU= 28.240
  10:59:42 STEP 1818 H= 454:30 +CPU= 28.623
  11:00:11 STEP 1819 H= 454:45 +CPU= 28.468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:01:04 STEP 1820 H= 455:00 +CPU= 30.137
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:01:57 STEP 1821 H= 455:15 +CPU= 30.323
  11:02:28 STEP 1822 H= 455:30 +CPU= 29.986
  11:02:59 STEP 1823 H= 455:45 +CPU= 30.098
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:04:45 STEP 1824 H= 456:00 +CPU= 41.446
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001824
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001824
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001824
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001728
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001728
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001728
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001740
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001740
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001740
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001752
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001752
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001752
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001764
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001764
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001764
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001776
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001776
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001776
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001788
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001788
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001788
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001800
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001800
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001800
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001812
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001812
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001812
Zipping up the intermediate file: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_a141_2016092300_20_1018_12289768_0_r1702435919_18.zip
Uploading the intermediate file: upload_file_18.zip
  11:05:19 STEP 1825 H= 456:15 +CPU= 32.415
  11:05:51 STEP 1826 H= 456:30 +CPU= 30.607
  11:06:22 STEP 1827 H= 456:45 +CPU= 29.924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:07:17 STEP 1828 H= 457:00 +CPU= 31.706
  11:07:46 STEP 1829 H= 457:15 +CPU= 28.702
  11:08:15 STEP 1830 H= 457:30 +CPU= 28.167
  11:08:44 STEP 1831 H= 457:45 +CPU= 29.077
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:09:38 STEP 1832 H= 458:00 +CPU= 31.283
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:10:31 STEP 1833 H= 458:15 +CPU= 30.422
  11:11:00 STEP 1834 H= 458:30 +CPU= 28.078
  11:11:28 STEP 1835 H= 458:45 +CPU= 27.978
  11:12:03 STEP 1836 H= 459:00 +CPU= 34.844
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001836
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001836
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001836
  11:12:32 STEP 1837 H= 459:15 +CPU= 27.956
  11:13:00 STEP 1838 H= 459:30 +CPU= 28.001
  11:13:37 STEP 1839 H= 459:45 +CPU= 35.517
  11:14:18 STEP 1840 H= 460:00 +CPU= 38.294
  11:14:57 STEP 1841 H= 460:15 +CPU= 37.461
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:16:00 STEP 1842 H= 460:30 +CPU= 37.302
  11:16:39 STEP 1843 H= 460:45 +CPU= 37.099
  11:17:18 STEP 1844 H= 461:00 +CPU= 37.915
  11:17:56 STEP 1845 H= 461:15 +CPU= 37.070
  11:18:34 STEP 1846 H= 461:30 +CPU= 37.130
  11:19:13 STEP 1847 H= 461:45 +CPU= 37.107
  11:20:01 STEP 1848 H= 462:00 +CPU= 46.788
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001848
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001848
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001848
  11:20:40 STEP 1849 H= 462:15 +CPU= 37.151
  11:21:19 STEP 1850 H= 462:30 +CPU= 37.145
  11:21:57 STEP 1851 H= 462:45 +CPU= 37.052
  11:22:36 STEP 1852 H= 463:00 +CPU= 37.850
  11:23:15 STEP 1853 H= 463:15 +CPU= 37.200
  11:23:53 STEP 1854 H= 463:30 +CPU= 37.193
  11:24:32 STEP 1855 H= 463:45 +CPU= 37.101
  11:25:12 STEP 1856 H= 464:00 +CPU= 37.965
  11:25:50 STEP 1857 H= 464:15 +CPU= 37.114
  11:26:29 STEP 1858 H= 464:30 +CPU= 37.096
  11:27:07 STEP 1859 H= 464:45 +CPU= 36.997
  11:27:55 STEP 1860 H= 465:00 +CPU= 46.489
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001860
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001860
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001860
  11:28:33 STEP 1861 H= 465:15 +CPU= 37.036
  11:29:11 STEP 1862 H= 465:30 +CPU= 37.009
  11:29:49 STEP 1863 H= 465:45 +CPU= 37.057
  11:30:29 STEP 1864 H= 466:00 +CPU= 37.848
  11:31:07 STEP 1865 H= 466:15 +CPU= 36.951
  11:31:45 STEP 1866 H= 466:30 +CPU= 37.043
  11:32:23 STEP 1867 H= 466:45 +CPU= 37.058
  11:33:02 STEP 1868 H= 467:00 +CPU= 37.799
  11:33:40 STEP 1869 H= 467:15 +CPU= 36.978
  11:34:18 STEP 1870 H= 467:30 +CPU= 37.035
  11:34:56 STEP 1871 H= 467:45 +CPU= 37.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:36:21 STEP 1872 H= 468:00 +CPU= 50.642
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001872
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001872
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001872
  11:36:59 STEP 1873 H= 468:15 +CPU= 37.045
  11:37:38 STEP 1874 H= 468:30 +CPU= 37.119
  11:38:16 STEP 1875 H= 468:45 +CPU= 37.017
  11:38:54 STEP 1876 H= 469:00 +CPU= 37.839
  11:39:33 STEP 1877 H= 469:15 +CPU= 37.128
  11:40:12 STEP 1878 H= 469:30 +CPU= 37.226
  11:40:51 STEP 1879 H= 469:45 +CPU= 37.085
  11:41:30 STEP 1880 H= 470:00 +CPU= 37.945
  11:42:08 STEP 1881 H= 470:15 +CPU= 37.232
  11:42:47 STEP 1882 H= 470:30 +CPU= 37.213
  11:43:25 STEP 1883 H= 470:45 +CPU= 37.180
  11:44:13 STEP 1884 H= 471:00 +CPU= 46.725
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001884
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001884
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001884
  11:44:52 STEP 1885 H= 471:15 +CPU= 37.242
  11:45:31 STEP 1886 H= 471:30 +CPU= 37.308
  11:46:10 STEP 1887 H= 471:45 +CPU= 37.225
  11:46:49 STEP 1888 H= 472:00 +CPU= 38.044
  11:47:27 STEP 1889 H= 472:15 +CPU= 37.174
  11:48:06 STEP 1890 H= 472:30 +CPU= 37.132
  11:48:44 STEP 1891 H= 472:45 +CPU= 37.115
  11:49:23 STEP 1892 H= 473:00 +CPU= 37.922
  11:50:02 STEP 1893 H= 473:15 +CPU= 37.294
  11:50:41 STEP 1894 H= 473:30 +CPU= 37.174
  11:51:20 STEP 1895 H= 473:45 +CPU= 37.200
  11:52:09 STEP 1896 H= 474:00 +CPU= 46.797
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001896
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001896
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001896
  11:52:47 STEP 1897 H= 474:15 +CPU= 37.145
  11:53:25 STEP 1898 H= 474:30 +CPU= 37.215
  11:54:04 STEP 1899 H= 474:45 +CPU= 37.192
  11:54:43 STEP 1900 H= 475:00 +CPU= 37.927
  11:55:22 STEP 1901 H= 475:15 +CPU= 37.357
  11:56:00 STEP 1902 H= 475:30 +CPU= 37.175
  11:56:39 STEP 1903 H= 475:45 +CPU= 37.232
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:58:15 STEP 1904 H= 476:00 +CPU= 33.754
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:59:50 STEP 1905 H= 476:15 +CPU= 30.970
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:00:46 STEP 1906 H= 476:30 +CPU= 31.601
  12:01:17 STEP 1907 H= 476:45 +CPU= 29.959
  12:01:55 STEP 1908 H= 477:00 +CPU= 37.166
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001908
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001908
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:03:11 STEP 1909 H= 477:15 +CPU= 31.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:47 STEP 1910 H= 477:30 +CPU= 31.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:06:42 STEP 1911 H= 477:45 +CPU= 30.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:37 STEP 1912 H= 478:00 +CPU= 30.728
  12:08:09 STEP 1913 H= 478:15 +CPU= 30.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:48 STEP 1914 H= 478:30 +CPU= 30.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:13:07 STEP 1915 H= 478:45 +CPU= 31.200
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:17:05 STEP 1916 H= 479:00 +CPU= 31.142
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:00 STEP 1917 H= 479:15 +CPU= 30.118
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:20:21 STEP 1918 H= 479:30 +CPU= 31.852
  12:20:53 STEP 1919 H= 479:45 +CPU= 29.744
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:22:18 STEP 1920 H= 480:00 +CPU= 38.583
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.8677917124E-21      I       0.2295993035E-07      I
ICloud fractionI      0.0000000000E+00      I       0.6018905739E-03      I
I  Relat. Hum. I      0.2623628041E+00      I       0.2927439916E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1210281589E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    319413892 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS798534.7 Mbytes/s, TOTAL TIME=    0.0085( 4.7%)
 12:22:18 STEP 1920 H= 480:00 +CPU= 38.583
 END CNT3
 NSTEP =  1920 CNT0   000000000  29.449  29.449  74.348   0.396               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         7150/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    128    0       46      1044068     7150 ************    26.2    13.9    12.2    14.7    26.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.89      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  261    6052.2    6052.2     10.24      0.00
   9 SPCM     - SPECTRAL COMP.                     240    1377.9    1377.9      2.14      0.00
  10 SCAN2M - PHYSICS                              241   42186.4   42186.4     65.93      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 20     975.8     975.8      0.13      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 11      66.8      66.8      0.00      0.00
  17 GRIDFPOS IN CNT4                               20      38.3      38.3      0.00      0.00
  27 RESTART FILES - WRITING                         5    3903.2    3903.2      0.13      0.00
  28 RESTART FILES - READING                         1   15553.6   15553.6      0.10      0.00
  29 SU4FPOS IN CNT4                                20       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                20    7918.2    7918.2      1.03      0.00
  31 POSDDH IN STEPO                                 2      43.7      43.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    241    8909.8    8909.8     13.92      0.00
  39 SU0YOMB                                         1     522.4     522.4      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    241     123.1     123.1      0.19      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              240     429.3     429.3      0.67      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              240      55.1      55.1      0.09      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 11       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      524     675.1     675.1      2.29      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       500     437.1     437.1      1.42      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        500     135.4     135.4      0.44      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       524     224.7     224.7      0.76      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1     135.5     135.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            524      57.7      57.7      0.20      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            500      31.4      31.4      0.10      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            524     112.0     112.0      0.38      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            500      37.9      37.9      0.12      0.00
 400 GSTATS                                      28864       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 26802       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    15421.5 CPU TIME    8448.4 VECTOR TIME     8448.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      112.2
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMGGh7zg+001920
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMSHh7zg+001920
Moving to projects directory: /mnt/WDred/boinc/slots/0/ICMUAh7zg+001920
Adding to the zip: /mnt/WDred/boinc/slots/0/NODE.001_01
Adding to the zip: /mnt/WDred/boinc/slots/0/ifs.stat
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001848
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001884
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001860
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001824
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001908
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001884
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001824
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001896
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001836
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001860
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001848
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001920
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001872
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001872
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001836
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001824
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001860
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001836
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001884
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001920
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001848
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMUAh7zg+001896
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001908
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001872
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001920
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMSHh7zg+001896
Adding to the zip: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_12289768/ICMGGh7zg+001908
Zipping up the final file: /mnt/WDred/boinc/projects/climateprediction.net/oifs_43r3_bl_a141_2016092300_20_1018_12289768_0_r1702435919_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
12:25:49 (14006): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
20 Jun 2024 10:28:51 1496581 22443355 oifs_43r3_bl_a141_2016092300_20_1018_12289768_0 1,727,100 67,683 0.0392
19 Jun 2024 20:30:40 1496581 22443355 oifs_43r3_bl_a141_2016092300_20_1018_12289768_0 1,296,000 51,699 0.0399


©2024 cpdn.org