climateprediction.net home page
Task 22315599

Task 22315599

Name oifs_43r3_00ma_2019110100_123_993_12214240_0
Workunit 12214240
Created 24 Feb 2023, 14:18:36 UTC
Sent 24 Feb 2023, 17:06:30 UTC
Report deadline 25 Apr 2023, 17:06:30 UTC
Received 28 Feb 2023, 16:50:25 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1478188
Run time 1 days 2 hours 30 min
CPU time 1 days 2 hours 30 min
Validate state Valid
Credit 14,873.04
Device peak FLOPS 4.00 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,406.71 MB
Peak swap size 4,984.00 MB
Peak disk usage 1,884.42 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064252:1677566572:0.001] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 1348MB (heap), 704MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x8102d0 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
double free or corruption (top)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3843] 1348 MB (maxheap), 704 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :       RERESF 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :        IOSTREAM_MIX:IO_GET 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.549] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :         IOSTREAM_MIX:READ_RECORD 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.550] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :          >MPL-IOSTREAM READ_RECORD ( 650) 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064609:1677566769:196.550] [c_drhook_print_@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] :           GRIB_API:IGRIB_READ_BYTES_REAL8 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064610:1677566770:197.550] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:5729:5729] [20230228:064610:1677566770:197.550] [signal_drhook@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8440 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE89FB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000810F01  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC8950  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC8522  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DF71F9  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DEFB93  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001ADE9C2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AD3B2A  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000009E193D  grib_api_interfac         483  grib_api_interface.F90
oifs_43r3_model.e  00000000009D4E7A  iostream_mix_mp_r        3400  iostream_mix.F90
oifs_43r3_model.e  00000000009CC3ED  iostream_mix_mp_i        2279  iostream_mix.F90
oifs_43r3_model.e  0000000000422A4D  reresf_                   359  reresf.F90
oifs_43r3_model.e  000000000041D1C8  cnt3_                     184  cnt3.F90
oifs_43r3_model.e  000000000041C8F2  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000407BD8  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040734D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9C50  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 00ma
(argv4) batchid: 993
(argv5) wuid: 12214240
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /data/boinc/boinc-client/slots/22
Project directory is: /data/boinc/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240
..mkdir for temp folder for results failed
Copying: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /data/boinc/boinc-client/slots/22/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /data/boinc/boinc-client/slots/22/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f924b6a2f740558a71cef6e190a09e37 to: /data/boinc/boinc-client/slots/22/oifs_43r3_00ma_2019110100_123_993_12214240.zip
Unzipping the namelist zip file: /data/boinc/boinc-client/slots/22/oifs_43r3_00ma_2019110100_123_993_12214240.zip
ic_ancil_file: ic_ancil_12214240
ifsdata_file: ifsdata_12214240
climate_data_file: clim_data_12214240
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4afc5af9d2fa21fe864c5f3a0175526b to: /data/boinc/boinc-client/slots/22/ic_ancil_12214240.zip
Unzipping the IC ancils zip file: /data/boinc/boinc-client/slots/22/ic_ancil_12214240.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eab791ab6daaa6e336927149a77f4443 to: /data/boinc/boinc-client/slots/22/ifsdata/ifsdata_12214240.zip
Unzipping the ifsdata_zip file: /data/boinc/boinc-client/slots/22/ifsdata/ifsdata_12214240.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /data/boinc/boinc-client/slots/22/159l_2/clim_data_12214240.zip
Unzipping the climate data zip file: /data/boinc/boinc-client/slots/22/159l_2/clim_data_12214240.zip
Checking for progress XML file: /data/boinc/boinc-client/slots/22/progress_file_12214240.xml
Opened progress file ok : /data/boinc/boinc-client/slots/22/progress_file_12214240.xml
-- Model is restarting --
Adjusting last_iter, 2593, to previous model restart step.
Creating progress file: /data/boinc/boinc-client/slots/22/progress_file_12214240.xml
last_cpu_time: 84119
upload_file_number: 108
last_iter: 2593
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268
The child process has been launched with process id: 18039
Executing the command: /data/boinc/boinc-client/slots/22/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:18039:18039] [20230228:071031:1677568231:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:11:41 STEP 2592 H=2592:00 +CPU= 38.185
  07:12:02 STEP 2593 H=2593:00 +CPU= 20.951
  07:12:23 STEP 2594 H=2594:00 +CPU= 20.689
  07:12:56 STEP 2595 H=2595:00 +CPU= 31.966
  07:13:18 STEP 2596 H=2596:00 +CPU= 21.965
  07:13:38 STEP 2597 H=2597:00 +CPU= 19.817
  07:14:11 STEP 2598 H=2598:00 +CPU= 32.824
  07:14:31 STEP 2599 H=2599:00 +CPU= 20.182
  07:14:51 STEP 2600 H=2600:00 +CPU= 19.668
  07:15:23 STEP 2601 H=2601:00 +CPU= 31.854
  07:15:44 STEP 2602 H=2602:00 +CPU= 20.509
  07:16:04 STEP 2603 H=2603:00 +CPU= 19.764
  07:16:40 STEP 2604 H=2604:00 +CPU= 35.520
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002604
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002604
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002604
  07:17:01 STEP 2605 H=2605:00 +CPU= 20.978
  07:17:24 STEP 2606 H=2606:00 +CPU= 23.044
  07:18:00 STEP 2607 H=2607:00 +CPU= 34.965
  07:18:23 STEP 2608 H=2608:00 +CPU= 22.827
  07:18:45 STEP 2609 H=2609:00 +CPU= 21.221
  07:19:20 STEP 2610 H=2610:00 +CPU= 34.697
  07:19:44 STEP 2611 H=2611:00 +CPU= 23.410
  07:20:08 STEP 2612 H=2612:00 +CPU= 23.453
  07:20:45 STEP 2613 H=2613:00 +CPU= 36.707
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:29:02 STEP 2614 H=2614:00 +CPU= 23.101
  07:29:24 STEP 2615 H=2615:00 +CPU= 21.256
  07:30:08 STEP 2616 H=2616:00 +CPU= 43.101
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002616
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002616
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002616
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002592
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002592
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002592
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002604
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002604
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002604
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_108.zip
Uploading the intermediate file: upload_file_108.zip
  07:30:31 STEP 2617 H=2617:00 +CPU= 22.825
  07:30:53 STEP 2618 H=2618:00 +CPU= 21.829
  07:31:27 STEP 2619 H=2619:00 +CPU= 33.630
  07:31:49 STEP 2620 H=2620:00 +CPU= 21.558
  07:32:10 STEP 2621 H=2621:00 +CPU= 20.966
  07:32:44 STEP 2622 H=2622:00 +CPU= 33.935
  07:33:05 STEP 2623 H=2623:00 +CPU= 20.856
  07:33:27 STEP 2624 H=2624:00 +CPU= 21.107
  07:34:00 STEP 2625 H=2625:00 +CPU= 33.212
  07:34:22 STEP 2626 H=2626:00 +CPU= 21.669
  07:34:42 STEP 2627 H=2627:00 +CPU= 20.008
  07:35:18 STEP 2628 H=2628:00 +CPU= 35.591
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002628
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002628
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002628
  07:35:39 STEP 2629 H=2629:00 +CPU= 20.645
  07:36:00 STEP 2630 H=2630:00 +CPU= 20.724
  07:36:33 STEP 2631 H=2631:00 +CPU= 33.070
  07:36:54 STEP 2632 H=2632:00 +CPU= 20.912
  07:37:14 STEP 2633 H=2633:00 +CPU= 19.926
  07:37:47 STEP 2634 H=2634:00 +CPU= 32.533
  07:38:08 STEP 2635 H=2635:00 +CPU= 20.746
  07:38:29 STEP 2636 H=2636:00 +CPU= 20.789
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:41:04 STEP 2637 H=2637:00 +CPU= 33.603
  07:41:26 STEP 2638 H=2638:00 +CPU= 21.920
  07:41:48 STEP 2639 H=2639:00 +CPU= 21.836
  07:42:28 STEP 2640 H=2640:00 +CPU= 39.577
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002640
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002640
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002640
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002616
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002616
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002616
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002628
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002628
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002628
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_109.zip
Uploading the intermediate file: upload_file_109.zip
  07:42:48 STEP 2641 H=2641:00 +CPU= 20.004
  07:43:10 STEP 2642 H=2642:00 +CPU= 21.740
  07:43:44 STEP 2643 H=2643:00 +CPU= 33.991
  07:44:06 STEP 2644 H=2644:00 +CPU= 21.705
  07:44:26 STEP 2645 H=2645:00 +CPU= 20.272
  07:45:01 STEP 2646 H=2646:00 +CPU= 34.428
  07:45:22 STEP 2647 H=2647:00 +CPU= 20.809
  07:45:44 STEP 2648 H=2648:00 +CPU= 21.496
  07:46:17 STEP 2649 H=2649:00 +CPU= 33.262
  07:46:40 STEP 2650 H=2650:00 +CPU= 21.951
  07:47:01 STEP 2651 H=2651:00 +CPU= 20.875
  07:47:37 STEP 2652 H=2652:00 +CPU= 35.940
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002652
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002652
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002652
  07:47:58 STEP 2653 H=2653:00 +CPU= 20.964
  07:48:20 STEP 2654 H=2654:00 +CPU= 21.856
  07:48:54 STEP 2655 H=2655:00 +CPU= 33.340
  07:49:15 STEP 2656 H=2656:00 +CPU= 21.349
  07:49:37 STEP 2657 H=2657:00 +CPU= 21.176
  07:50:12 STEP 2658 H=2658:00 +CPU= 35.191
  07:50:34 STEP 2659 H=2659:00 +CPU= 21.426
  07:50:56 STEP 2660 H=2660:00 +CPU= 22.003
  07:51:30 STEP 2661 H=2661:00 +CPU= 33.510
  07:51:52 STEP 2662 H=2662:00 +CPU= 21.871
  07:52:15 STEP 2663 H=2663:00 +CPU= 22.740
  07:53:04 STEP 2664 H=2664:00 +CPU= 46.694
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002664
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002664
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002664
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002640
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002640
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002640
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002652
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002652
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002652
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_110.zip
Uploading the intermediate file: upload_file_110.zip
  07:53:31 STEP 2665 H=2665:00 +CPU= 27.054
  07:53:57 STEP 2666 H=2666:00 +CPU= 25.530
  07:54:40 STEP 2667 H=2667:00 +CPU= 41.991
  07:55:07 STEP 2668 H=2668:00 +CPU= 25.890
  07:55:33 STEP 2669 H=2669:00 +CPU= 25.799
  07:56:16 STEP 2670 H=2670:00 +CPU= 42.459
  07:56:43 STEP 2671 H=2671:00 +CPU= 26.336
  07:57:11 STEP 2672 H=2672:00 +CPU= 27.008
  07:57:54 STEP 2673 H=2673:00 +CPU= 42.799
  07:58:20 STEP 2674 H=2674:00 +CPU= 24.840
  07:58:44 STEP 2675 H=2675:00 +CPU= 23.827
  07:59:27 STEP 2676 H=2676:00 +CPU= 42.344
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002676
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002676
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002676
  07:59:54 STEP 2677 H=2677:00 +CPU= 26.725
  08:00:26 STEP 2678 H=2678:00 +CPU= 31.333
  08:01:09 STEP 2679 H=2679:00 +CPU= 42.361
  08:01:38 STEP 2680 H=2680:00 +CPU= 28.474
  08:02:02 STEP 2681 H=2681:00 +CPU= 23.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:29:40 STEP 2682 H=2682:00 +CPU= 47.176
  08:30:11 STEP 2683 H=2683:00 +CPU= 28.108
  08:30:43 STEP 2684 H=2684:00 +CPU= 29.548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:36:13 STEP 2685 H=2685:00 +CPU= 46.490
  08:36:46 STEP 2686 H=2686:00 +CPU= 30.240
  08:37:19 STEP 2687 H=2687:00 +CPU= 30.248
  08:38:22 STEP 2688 H=2688:00 +CPU= 55.650
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002688
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002688
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002688
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002664
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002664
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002664
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002676
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002676
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002676
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_111.zip
Uploading the intermediate file: upload_file_111.zip
  08:38:54 STEP 2689 H=2689:00 +CPU= 29.963
  08:39:27 STEP 2690 H=2690:00 +CPU= 30.405
  08:40:19 STEP 2691 H=2691:00 +CPU= 48.254
  08:40:51 STEP 2692 H=2692:00 +CPU= 30.177
  08:41:23 STEP 2693 H=2693:00 +CPU= 29.512
  08:42:15 STEP 2694 H=2694:00 +CPU= 48.349
  08:42:47 STEP 2695 H=2695:00 +CPU= 29.359
  08:43:20 STEP 2696 H=2696:00 +CPU= 30.459
  08:44:10 STEP 2697 H=2697:00 +CPU= 46.813
  08:44:42 STEP 2698 H=2698:00 +CPU= 29.920
  08:45:14 STEP 2699 H=2699:00 +CPU= 29.722
  08:46:09 STEP 2700 H=2700:00 +CPU= 50.992
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002700
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002700
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002700
  08:46:42 STEP 2701 H=2701:00 +CPU= 30.511
  08:47:15 STEP 2702 H=2702:00 +CPU= 31.100
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:50:39 STEP 2703 H=2703:00 +CPU= 48.721
  08:51:11 STEP 2704 H=2704:00 +CPU= 29.701
  08:51:42 STEP 2705 H=2705:00 +CPU= 29.792
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:55:06 STEP 2706 H=2706:00 +CPU= 48.673
  08:55:38 STEP 2707 H=2707:00 +CPU= 30.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:45:03 STEP 2708 H=2708:00 +CPU= 29.530
  09:45:50 STEP 2709 H=2709:00 +CPU= 45.783
  09:46:18 STEP 2710 H=2710:00 +CPU= 27.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:17:08 STEP 2711 H=2711:00 +CPU= 27.987
  10:18:08 STEP 2712 H=2712:00 +CPU= 52.115
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002712
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002712
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002712
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002688
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002688
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002688
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002700
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002700
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002700
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_112.zip
Uploading the intermediate file: upload_file_112.zip
  10:18:36 STEP 2713 H=2713:00 +CPU= 26.345
  10:19:05 STEP 2714 H=2714:00 +CPU= 27.266
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:42:18 STEP 2715 H=2715:00 +CPU= 45.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:01:27 STEP 2716 H=2716:00 +CPU= 29.470
  11:01:52 STEP 2717 H=2717:00 +CPU= 24.650
  11:02:31 STEP 2718 H=2718:00 +CPU= 37.957
  11:02:55 STEP 2719 H=2719:00 +CPU= 23.197
  11:03:20 STEP 2720 H=2720:00 +CPU= 24.283
  11:03:59 STEP 2721 H=2721:00 +CPU= 38.549
  11:04:23 STEP 2722 H=2722:00 +CPU= 22.791
  11:04:48 STEP 2723 H=2723:00 +CPU= 24.364
  11:05:37 STEP 2724 H=2724:00 +CPU= 47.783
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002724
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002724
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002724
  11:06:02 STEP 2725 H=2725:00 +CPU= 23.914
  11:06:27 STEP 2726 H=2726:00 +CPU= 24.428
  11:07:06 STEP 2727 H=2727:00 +CPU= 38.031
  11:07:32 STEP 2728 H=2728:00 +CPU= 25.487
  11:07:58 STEP 2729 H=2729:00 +CPU= 25.201
  11:08:41 STEP 2730 H=2730:00 +CPU= 42.485
  11:09:05 STEP 2731 H=2731:00 +CPU= 23.156
  11:09:30 STEP 2732 H=2732:00 +CPU= 23.683
  11:10:09 STEP 2733 H=2733:00 +CPU= 38.610
  11:10:34 STEP 2734 H=2734:00 +CPU= 23.512
  11:10:58 STEP 2735 H=2735:00 +CPU= 23.443
  11:11:50 STEP 2736 H=2736:00 +CPU= 48.459
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002736
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002736
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002736
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002712
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002712
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002712
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002724
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002724
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002724
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_113.zip
Uploading the intermediate file: upload_file_113.zip
  11:12:17 STEP 2737 H=2737:00 +CPU= 26.934
  11:12:46 STEP 2738 H=2738:00 +CPU= 27.627
  11:13:32 STEP 2739 H=2739:00 +CPU= 44.248
  11:14:01 STEP 2740 H=2740:00 +CPU= 28.029
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:22:37 STEP 2741 H=2741:00 +CPU= 31.690
  12:23:30 STEP 2742 H=2742:00 +CPU= 47.885
  12:24:02 STEP 2743 H=2743:00 +CPU= 30.143
  12:24:35 STEP 2744 H=2744:00 +CPU= 30.086
  12:25:26 STEP 2745 H=2745:00 +CPU= 47.300
  12:25:56 STEP 2746 H=2746:00 +CPU= 27.507
  12:26:27 STEP 2747 H=2747:00 +CPU= 28.885
  12:27:23 STEP 2748 H=2748:00 +CPU= 52.217
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002748
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002748
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002748
  12:27:54 STEP 2749 H=2749:00 +CPU= 28.272
  12:28:25 STEP 2750 H=2750:00 +CPU= 28.917
  12:29:13 STEP 2751 H=2751:00 +CPU= 43.692
  12:29:45 STEP 2752 H=2752:00 +CPU= 29.776
  12:30:16 STEP 2753 H=2753:00 +CPU= 28.900
  12:31:07 STEP 2754 H=2754:00 +CPU= 46.601
  12:31:38 STEP 2755 H=2755:00 +CPU= 29.290
  12:32:10 STEP 2756 H=2756:00 +CPU= 29.109
  12:33:00 STEP 2757 H=2757:00 +CPU= 46.533
  12:33:32 STEP 2758 H=2758:00 +CPU= 29.594
  12:34:04 STEP 2759 H=2759:00 +CPU= 29.087
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:11:06 STEP 2760 H=2760:00 +CPU= 51.855
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002760
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002760
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002760
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002736
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002736
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002736
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002748
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002748
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002748
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_114.zip
Uploading the intermediate file: upload_file_114.zip
  14:11:35 STEP 2761 H=2761:00 +CPU= 28.496
  14:12:04 STEP 2762 H=2762:00 +CPU= 28.482
  14:12:50 STEP 2763 H=2763:00 +CPU= 44.249
  14:13:16 STEP 2764 H=2764:00 +CPU= 26.374
  14:13:41 STEP 2765 H=2765:00 +CPU= 24.141
  14:14:26 STEP 2766 H=2766:00 +CPU= 44.144
  14:14:53 STEP 2767 H=2767:00 +CPU= 26.382
  14:15:21 STEP 2768 H=2768:00 +CPU= 27.030
  14:16:06 STEP 2769 H=2769:00 +CPU= 45.052
  14:16:34 STEP 2770 H=2770:00 +CPU= 26.969
  14:16:58 STEP 2771 H=2771:00 +CPU= 23.300
  14:17:43 STEP 2772 H=2772:00 +CPU= 44.314
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002772
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002772
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002772
  14:18:09 STEP 2773 H=2773:00 +CPU= 25.730
  14:18:38 STEP 2774 H=2774:00 +CPU= 27.688
  14:19:21 STEP 2775 H=2775:00 +CPU= 42.236
  14:19:46 STEP 2776 H=2776:00 +CPU= 25.070
  14:20:14 STEP 2777 H=2777:00 +CPU= 27.399
  14:20:57 STEP 2778 H=2778:00 +CPU= 41.557
  14:21:25 STEP 2779 H=2779:00 +CPU= 27.437
  14:21:53 STEP 2780 H=2780:00 +CPU= 27.649
  14:22:37 STEP 2781 H=2781:00 +CPU= 43.366
  14:23:00 STEP 2782 H=2782:00 +CPU= 22.913
  14:23:24 STEP 2783 H=2783:00 +CPU= 23.232
  14:24:17 STEP 2784 H=2784:00 +CPU= 49.294
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002784
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002784
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002784
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002760
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002760
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002760
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002772
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002772
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002772
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_115.zip
Uploading the intermediate file: upload_file_115.zip
  14:24:42 STEP 2785 H=2785:00 +CPU= 24.709
  14:25:07 STEP 2786 H=2786:00 +CPU= 23.947
  14:25:44 STEP 2787 H=2787:00 +CPU= 36.411
  14:26:08 STEP 2788 H=2788:00 +CPU= 23.678
  14:26:33 STEP 2789 H=2789:00 +CPU= 23.803
  14:27:12 STEP 2790 H=2790:00 +CPU= 38.731
  14:27:39 STEP 2791 H=2791:00 +CPU= 26.290
  14:28:06 STEP 2792 H=2792:00 +CPU= 26.823
  14:28:46 STEP 2793 H=2793:00 +CPU= 38.831
  14:29:12 STEP 2794 H=2794:00 +CPU= 25.091
  14:29:34 STEP 2795 H=2795:00 +CPU= 22.225
  14:30:18 STEP 2796 H=2796:00 +CPU= 42.465
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002796
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002796
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002796
  14:30:42 STEP 2797 H=2797:00 +CPU= 23.313
  14:31:06 STEP 2798 H=2798:00 +CPU= 23.747
  14:31:44 STEP 2799 H=2799:00 +CPU= 37.535
  14:32:11 STEP 2800 H=2800:00 +CPU= 26.126
  14:32:37 STEP 2801 H=2801:00 +CPU= 24.871
  14:33:16 STEP 2802 H=2802:00 +CPU= 38.782
  14:33:40 STEP 2803 H=2803:00 +CPU= 23.178
  14:34:03 STEP 2804 H=2804:00 +CPU= 22.746
  14:34:44 STEP 2805 H=2805:00 +CPU= 39.606
  14:35:07 STEP 2806 H=2806:00 +CPU= 23.289
  14:35:32 STEP 2807 H=2807:00 +CPU= 24.165
  14:36:22 STEP 2808 H=2808:00 +CPU= 47.152
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002808
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002808
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002808
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002784
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002784
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002784
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002796
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002796
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002796
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_116.zip
Uploading the intermediate file: upload_file_116.zip
  14:36:44 STEP 2809 H=2809:00 +CPU= 21.878
  14:37:11 STEP 2810 H=2810:00 +CPU= 25.957
  14:37:55 STEP 2811 H=2811:00 +CPU= 42.632
  14:38:23 STEP 2812 H=2812:00 +CPU= 27.393
  14:38:47 STEP 2813 H=2813:00 +CPU= 23.697
  14:39:26 STEP 2814 H=2814:00 +CPU= 37.584
  14:39:51 STEP 2815 H=2815:00 +CPU= 24.882
  14:40:15 STEP 2816 H=2816:00 +CPU= 23.471
  14:40:57 STEP 2817 H=2817:00 +CPU= 40.569
  14:41:22 STEP 2818 H=2818:00 +CPU= 24.053
  14:41:49 STEP 2819 H=2819:00 +CPU= 26.592
  14:42:35 STEP 2820 H=2820:00 +CPU= 45.401
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002820
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002820
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002820
  14:43:01 STEP 2821 H=2821:00 +CPU= 24.485
  14:43:26 STEP 2822 H=2822:00 +CPU= 24.991
  14:44:07 STEP 2823 H=2823:00 +CPU= 39.599
  14:44:33 STEP 2824 H=2824:00 +CPU= 25.834
  14:44:59 STEP 2825 H=2825:00 +CPU= 24.943
  14:45:38 STEP 2826 H=2826:00 +CPU= 38.667
  14:46:08 STEP 2827 H=2827:00 +CPU= 28.702
  14:46:37 STEP 2828 H=2828:00 +CPU= 28.710
  14:47:19 STEP 2829 H=2829:00 +CPU= 41.231
  14:47:44 STEP 2830 H=2830:00 +CPU= 24.532
  14:48:12 STEP 2831 H=2831:00 +CPU= 26.899
  14:49:10 STEP 2832 H=2832:00 +CPU= 55.519
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002832
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002832
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002832
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002808
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002808
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002808
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002820
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002820
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002820
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_117.zip
Uploading the intermediate file: upload_file_117.zip
Uploading trickle at timestep: 10195200
  14:49:40 STEP 2833 H=2833:00 +CPU= 29.642
  14:50:10 STEP 2834 H=2834:00 +CPU= 29.054
  14:50:54 STEP 2835 H=2835:00 +CPU= 43.052
  14:51:21 STEP 2836 H=2836:00 +CPU= 26.238
  14:51:51 STEP 2837 H=2837:00 +CPU= 29.347
  14:52:35 STEP 2838 H=2838:00 +CPU= 42.825
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  15:23:26 STEP 2839 H=2839:00 +CPU= 28.322
  15:24:02 STEP 2840 H=2840:00 +CPU= 31.438
  15:24:46 STEP 2841 H=2841:00 +CPU= 43.278
  15:25:13 STEP 2842 H=2842:00 +CPU= 26.374
  15:25:43 STEP 2843 H=2843:00 +CPU= 28.455
  15:26:33 STEP 2844 H=2844:00 +CPU= 48.297
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002844
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002844
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002844
  15:27:01 STEP 2845 H=2845:00 +CPU= 27.018
  15:27:29 STEP 2846 H=2846:00 +CPU= 27.174
  15:28:11 STEP 2847 H=2847:00 +CPU= 41.143
  15:28:39 STEP 2848 H=2848:00 +CPU= 26.808
  15:29:06 STEP 2849 H=2849:00 +CPU= 26.326
  15:29:49 STEP 2850 H=2850:00 +CPU= 42.178
  15:30:17 STEP 2851 H=2851:00 +CPU= 27.314
  15:30:45 STEP 2852 H=2852:00 +CPU= 27.466
  15:31:28 STEP 2853 H=2853:00 +CPU= 41.345
  15:31:57 STEP 2854 H=2854:00 +CPU= 28.779
  15:32:25 STEP 2855 H=2855:00 +CPU= 26.384
  15:33:20 STEP 2856 H=2856:00 +CPU= 50.209
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002856
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002856
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002856
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002832
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002832
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002832
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002844
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002844
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002844
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_118.zip
Uploading the intermediate file: upload_file_118.zip
  15:33:46 STEP 2857 H=2857:00 +CPU= 25.380
  15:34:12 STEP 2858 H=2858:00 +CPU= 25.807
  15:34:58 STEP 2859 H=2859:00 +CPU= 44.612
  15:35:27 STEP 2860 H=2860:00 +CPU= 28.560
  15:35:56 STEP 2861 H=2861:00 +CPU= 28.058
  15:36:43 STEP 2862 H=2862:00 +CPU= 45.980
  15:37:08 STEP 2863 H=2863:00 +CPU= 24.054
  15:37:37 STEP 2864 H=2864:00 +CPU= 27.878
  15:38:22 STEP 2865 H=2865:00 +CPU= 43.994
  15:38:51 STEP 2866 H=2866:00 +CPU= 28.591
  15:39:20 STEP 2867 H=2867:00 +CPU= 28.232
  15:40:08 STEP 2868 H=2868:00 +CPU= 46.985
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002868
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002868
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002868
  15:40:37 STEP 2869 H=2869:00 +CPU= 28.329
  15:41:03 STEP 2870 H=2870:00 +CPU= 25.656
  15:41:42 STEP 2871 H=2871:00 +CPU= 37.662
  15:42:07 STEP 2872 H=2872:00 +CPU= 25.003
  15:42:34 STEP 2873 H=2873:00 +CPU= 26.423
  15:43:18 STEP 2874 H=2874:00 +CPU= 41.880
  15:43:45 STEP 2875 H=2875:00 +CPU= 26.602
  15:44:14 STEP 2876 H=2876:00 +CPU= 27.887
  15:44:58 STEP 2877 H=2877:00 +CPU= 43.624
  15:45:22 STEP 2878 H=2878:00 +CPU= 23.000
  15:45:49 STEP 2879 H=2879:00 +CPU= 26.473
  15:46:39 STEP 2880 H=2880:00 +CPU= 47.094
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002880
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002880
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002880
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002856
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002856
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002856
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002868
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002868
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002868
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_119.zip
Uploading the intermediate file: upload_file_119.zip
  15:47:04 STEP 2881 H=2881:00 +CPU= 24.594
  15:47:32 STEP 2882 H=2882:00 +CPU= 26.814
  15:48:12 STEP 2883 H=2883:00 +CPU= 39.403
  15:48:39 STEP 2884 H=2884:00 +CPU= 25.858
  15:49:04 STEP 2885 H=2885:00 +CPU= 24.341
  15:49:45 STEP 2886 H=2886:00 +CPU= 40.360
  15:50:09 STEP 2887 H=2887:00 +CPU= 23.353
  15:50:37 STEP 2888 H=2888:00 +CPU= 27.619
  15:51:16 STEP 2889 H=2889:00 +CPU= 37.710
  15:51:42 STEP 2890 H=2890:00 +CPU= 25.207
  15:52:08 STEP 2891 H=2891:00 +CPU= 25.585
  15:52:56 STEP 2892 H=2892:00 +CPU= 46.231
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002892
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002892
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002892
  15:53:25 STEP 2893 H=2893:00 +CPU= 28.941
  15:53:53 STEP 2894 H=2894:00 +CPU= 27.510
  15:54:36 STEP 2895 H=2895:00 +CPU= 41.706
  15:55:04 STEP 2896 H=2896:00 +CPU= 27.686
  15:55:31 STEP 2897 H=2897:00 +CPU= 25.979
  15:56:16 STEP 2898 H=2898:00 +CPU= 44.328
  15:56:42 STEP 2899 H=2899:00 +CPU= 25.695
  15:57:08 STEP 2900 H=2900:00 +CPU= 25.176
  15:57:51 STEP 2901 H=2901:00 +CPU= 41.505
  15:58:18 STEP 2902 H=2902:00 +CPU= 26.996
  15:58:47 STEP 2903 H=2903:00 +CPU= 27.647
  15:59:42 STEP 2904 H=2904:00 +CPU= 52.991
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002904
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002904
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002904
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002880
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002880
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002880
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002892
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002892
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002892
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_120.zip
Uploading the intermediate file: upload_file_120.zip
  16:00:10 STEP 2905 H=2905:00 +CPU= 26.901
  16:00:37 STEP 2906 H=2906:00 +CPU= 25.964
  16:01:19 STEP 2907 H=2907:00 +CPU= 41.054
  16:01:47 STEP 2908 H=2908:00 +CPU= 27.730
  16:02:15 STEP 2909 H=2909:00 +CPU= 27.767
  16:03:00 STEP 2910 H=2910:00 +CPU= 43.764
  16:03:29 STEP 2911 H=2911:00 +CPU= 27.567
  16:03:58 STEP 2912 H=2912:00 +CPU= 28.173
  16:04:41 STEP 2913 H=2913:00 +CPU= 42.547
  16:05:10 STEP 2914 H=2914:00 +CPU= 28.310
  16:05:36 STEP 2915 H=2915:00 +CPU= 25.052
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:18:55 STEP 2916 H=2916:00 +CPU= 43.770
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002916
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002916
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002916
  16:19:22 STEP 2917 H=2917:00 +CPU= 26.358
  16:19:47 STEP 2918 H=2918:00 +CPU= 24.488
  16:20:33 STEP 2919 H=2919:00 +CPU= 44.192
  16:21:01 STEP 2920 H=2920:00 +CPU= 27.577
  16:21:27 STEP 2921 H=2921:00 +CPU= 25.578
  16:22:08 STEP 2922 H=2922:00 +CPU= 39.527
  16:22:34 STEP 2923 H=2923:00 +CPU= 26.043
  16:23:01 STEP 2924 H=2924:00 +CPU= 25.748
  16:23:47 STEP 2925 H=2925:00 +CPU= 44.729
  16:24:14 STEP 2926 H=2926:00 +CPU= 27.046
  16:24:41 STEP 2927 H=2927:00 +CPU= 25.987
  16:25:35 STEP 2928 H=2928:00 +CPU= 49.956
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002928
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002928
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002928
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002904
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002904
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002904
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002916
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002916
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002916
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_121.zip
Uploading the intermediate file: upload_file_121.zip
  16:26:03 STEP 2929 H=2929:00 +CPU= 27.056
  16:26:32 STEP 2930 H=2930:00 +CPU= 28.283
  16:27:16 STEP 2931 H=2931:00 +CPU= 43.306
  16:27:45 STEP 2932 H=2932:00 +CPU= 28.510
  16:28:13 STEP 2933 H=2933:00 +CPU= 28.021
  16:28:58 STEP 2934 H=2934:00 +CPU= 43.268
  16:29:23 STEP 2935 H=2935:00 +CPU= 24.676
  16:29:52 STEP 2936 H=2936:00 +CPU= 27.860
  16:30:33 STEP 2937 H=2937:00 +CPU= 40.844
  16:31:04 STEP 2938 H=2938:00 +CPU= 30.229
  16:31:31 STEP 2939 H=2939:00 +CPU= 26.399
  16:32:17 STEP 2940 H=2940:00 +CPU= 44.697
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002940
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002940
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002940
  16:32:39 STEP 2941 H=2941:00 +CPU= 21.988
  16:33:08 STEP 2942 H=2942:00 +CPU= 27.805
  16:33:55 STEP 2943 H=2943:00 +CPU= 46.532
  16:34:21 STEP 2944 H=2944:00 +CPU= 25.519
  16:34:48 STEP 2945 H=2945:00 +CPU= 26.538
  16:35:35 STEP 2946 H=2946:00 +CPU= 45.454
  16:36:00 STEP 2947 H=2947:00 +CPU= 25.104
  16:36:26 STEP 2948 H=2948:00 +CPU= 25.285
  16:37:09 STEP 2949 H=2949:00 +CPU= 42.200
  16:37:39 STEP 2950 H=2950:00 +CPU= 29.450
  16:38:07 STEP 2951 H=2951:00 +CPU= 26.922
  16:39:04 STEP 2952 H=2952:00 +CPU= 55.143
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.14294232989170     
  MAXGPFV : MAX. VALUE =   0.765037370192713     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    333707978 BYTES IN      8 RECORDS TRANSFERRED IN    0.0016 SECONDS208567.5 Mbytes/s, TOTAL TIME=    0.0186( 8.6%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 16:39:04 STEP 2952 H=2952:00 +CPU= 55.143
 END CNT3
 NSTEP =  2952 CNT0   000000000  48.778  48.778  49.822   0.979               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         5595/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    138    0       35       786125     5595 ************  5831.1    25.9  5805.3    26.6  5832.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.92      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  391    2754.6    2754.6      3.16      0.00
   9 SPCM     - SPECTRAL COMP.                     360     359.7     359.7      0.38      0.00
  10 SCAN2M - PHYSICS                              361   34926.9   34926.9     36.96      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 30  194412.3  194412.3     17.10      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 16      37.4      37.4      0.00      0.00
  13 SCAN2M - RADIATION CALC.                      121   19142.0   19142.0      6.79      0.00
  17 GRIDFPOS IN CNT4                               30      64.2      64.2      0.01      0.00
  27 RESTART FILES - WRITING                        15  388817.6  388817.6     17.10      0.00
  28 RESTART FILES - READING                         1   27115.7   27115.7      0.08      0.00
  29 SU4FPOS IN CNT4                                30       6.4       6.4      0.00      0.00
  30 DYNFPOS IN CNT4                                30    2837.6    2837.6      0.25      0.00
  31 POSDDH IN STEPO                                 2      15.5      15.5      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    361   15992.2   15992.2     16.92      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL           361    6796.1    6796.1      7.19      0.00
  39 SU0YOMB                                         1     363.1     363.1      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    361      20.9      20.9      0.02      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              360     204.9     204.9      0.22      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              360     465.8     465.8      0.49      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 16       2.8       2.8      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1236    1115.5    1115.5      4.04      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       750    2595.9    2595.9      5.71      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        750      75.5      75.5      0.17      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1236      53.6      53.6      0.19      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      18.6      18.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1236      30.2      30.2      0.11      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            750      23.8      23.8      0.05      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1236      28.2      28.2      0.10      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            750      11.8      11.8      0.03      0.00
 400 GSTATS                                      72820       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 69682       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    34112.9 CPU TIME   11281.1 VECTOR TIME    11281.1


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      311.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMGGhnw4+002952
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMSHhnw4+002952
Moving to projects directory: /data/boinc/boinc-client/slots/22/ICMUAhnw4+002952
Adding to the zip: /data/boinc/boinc-client/slots/22/NODE.001_01
Adding to the zip: /data/boinc/boinc-client/slots/22/ifs.stat
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002940
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002952
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002928
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002940
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002928
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002928
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMGGhnw4+002940
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMUAhnw4+002952
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_12214240/ICMSHhnw4+002952
Zipping up the final file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_00ma_2019110100_123_993_12214240_0_r1498665268_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
16:42:28 (17505): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:55:00 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 10,623,600 95,400 0.0090
09 May 2023 16:54:59 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 10,195,200 91,493 0.0090
09 May 2023 16:54:51 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 8,899,200 80,095 0.0090
09 May 2023 16:54:50 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 8,035,200 72,921 0.0091
09 May 2023 16:54:48 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 7,171,200 65,785 0.0092
09 May 2023 16:54:36 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 5,875,200 53,952 0.0092
09 May 2023 16:54:31 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 5,011,200 46,495 0.0093
09 May 2023 16:54:02 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 3,542,400 32,871 0.0093
09 May 2023 16:53:54 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 2,678,400 24,943 0.0093
09 May 2023 16:53:49 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 1,814,400 17,114 0.0094
09 May 2023 16:52:56 1478188 22315599 oifs_43r3_00ma_2019110100_123_993_12214240_0 864,000 7,959 0.0092


©2024 cpdn.org