climateprediction.net home page
Task 22443545

Task 22443545

Name oifs_43r3_bl_a199_2016092300_20_1018_12289956_0
Workunit 12289956
Created 12 Jun 2024, 22:19:11 UTC
Sent 14 Jun 2024, 2:04:20 UTC
Report deadline 13 Aug 2024, 2:04:20 UTC
Received 15 Jun 2024, 12:10:16 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1549270
Run time 15 hours 38 min 45 sec
CPU time 15 hours 16 min 12 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 3.80 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,015.75 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,354.05 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
 H= 372:30 +CPU= 29.334
  20:54:39 STEP 1491 H= 372:45 +CPU= 29.937
  20:55:11 STEP 1492 H= 373:00 +CPU= 30.476
  20:55:42 STEP 1493 H= 373:15 +CPU= 29.652
  20:56:14 STEP 1494 H= 373:30 +CPU= 29.738
  20:56:45 STEP 1495 H= 373:45 +CPU= 29.193
  20:57:16 STEP 1496 H= 374:00 +CPU= 29.700
  20:57:46 STEP 1497 H= 374:15 +CPU= 28.959
  20:58:16 STEP 1498 H= 374:30 +CPU= 29.107
  20:58:46 STEP 1499 H= 374:45 +CPU= 28.964
  20:59:23 STEP 1500 H= 375:00 +CPU= 36.254
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001500
  20:59:53 STEP 1501 H= 375:15 +CPU= 28.845
  21:00:23 STEP 1502 H= 375:30 +CPU= 29.147
  21:00:52 STEP 1503 H= 375:45 +CPU= 28.893
  21:01:25 STEP 1504 H= 376:00 +CPU= 30.962
  21:01:55 STEP 1505 H= 376:15 +CPU= 29.023
  21:02:25 STEP 1506 H= 376:30 +CPU= 28.733
  21:02:54 STEP 1507 H= 376:45 +CPU= 28.682
  21:03:24 STEP 1508 H= 377:00 +CPU= 28.719
  21:03:54 STEP 1509 H= 377:15 +CPU= 29.386
  21:04:23 STEP 1510 H= 377:30 +CPU= 28.664
  21:04:51 STEP 1511 H= 377:45 +CPU= 27.444
  21:05:25 STEP 1512 H= 378:00 +CPU= 34.125
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001512
  21:05:53 STEP 1513 H= 378:15 +CPU= 27.749
  21:06:21 STEP 1514 H= 378:30 +CPU= 27.675
  21:06:52 STEP 1515 H= 378:45 +CPU= 29.003
  21:07:21 STEP 1516 H= 379:00 +CPU= 29.178
  21:07:51 STEP 1517 H= 379:15 +CPU= 28.573
  21:08:20 STEP 1518 H= 379:30 +CPU= 28.436
  21:08:49 STEP 1519 H= 379:45 +CPU= 28.911
  21:09:19 STEP 1520 H= 380:00 +CPU= 28.888
  21:09:48 STEP 1521 H= 380:15 +CPU= 28.743
  21:10:18 STEP 1522 H= 380:30 +CPU= 28.863
  21:10:47 STEP 1523 H= 380:45 +CPU= 28.426
  21:11:23 STEP 1524 H= 381:00 +CPU= 35.210
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001524
  21:11:52 STEP 1525 H= 381:15 +CPU= 28.694
  21:12:21 STEP 1526 H= 381:30 +CPU= 28.475
  21:12:50 STEP 1527 H= 381:45 +CPU= 28.390
  21:13:20 STEP 1528 H= 382:00 +CPU= 29.329
  21:13:52 STEP 1529 H= 382:15 +CPU= 30.110
  21:14:24 STEP 1530 H= 382:30 +CPU= 30.029
  21:14:56 STEP 1531 H= 382:45 +CPU= 30.417
  21:15:27 STEP 1532 H= 383:00 +CPU= 30.333
  21:15:58 STEP 1533 H= 383:15 +CPU= 29.759
  21:16:28 STEP 1534 H= 383:30 +CPU= 28.829
  21:17:00 STEP 1535 H= 383:45 +CPU= 30.137
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.059] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2554MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.061] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.061] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.061] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.061] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.061] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 2679 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         TRANSDIRH 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          TRANSDIR_MDL 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           DIR_TRANS 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            >TRS-LTDIR_CTL   - DIRECT ( 103) 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-LTDIR_CTL   - DIRECT (1645) 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              LTDIR_MOD 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212632:1718418392:45408.062] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               LE_DGEMM 
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212633:1718418393:45409.062] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Funk2:1:1:2464:2464] [20240614:212633:1718418393:45409.062] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001783BE4  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000B40CE8  ledir_mod_mp_ledi         153  ledir_mod.F90
oifs_43r3_model.e  0000000000B3AEB3  ltdir_mod_mp_ltdi         151  ltdir_mod.F90
oifs_43r3_model.e  0000000000B3A388  ltdir_ctl_mod_mp_          84  ltdir_ctl_mod.F90
oifs_43r3_model.e  0000000000B39EF8  dir_trans_ctl_mod         176  dir_trans_ctl_mod.F90
oifs_43r3_model.e  0000000000857BE1  dir_trans_                488  dir_trans.F90
oifs_43r3_model.e  000000000132E731  transdir_mdl_             148  transdir_mdl.F90
oifs_43r3_model.e  000000000132CB90  transdirh_                 56  transdirh.F90
oifs_43r3_model.e  00000000006882B2  stepo_                    375  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a199
(argv4) batchid: 1018
(argv5) wuid: 12289956
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_26ac3dc62d7bc69dced7ff1b0af3e9bb to: /var/lib/boinc-client/slots/4/oifs_43r3_bl_a199_2016092300_20_1018_12289956.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_bl_a199_2016092300_20_1018_12289956.zip
ic_ancil_file: ic_ancil_12289956
ifsdata_file: ifsdata_12289956
climate_data_file: clim_data_12289956
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/4/ic_ancil_12289956.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12289956.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_282af2f0d036bd9ba5145dd1011cf07f to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12289956.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12289956.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12289956.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12289956.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12289956.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12289956.xml
-- Model is restarting --
Adjusting last_iter, 1536, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12289956.xml
last_cpu_time: 43949
upload_file_number: 15
last_iter: 1537
last_upload: 1296000
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a199_2016092300_20_1018_12289956_0_r1106086708
The child process has been launched with process id: 2434
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Funk2:1:1:2434:2434] [20240615:030051:1718438451:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a199_2016092300_20_1018_12289956_0_r1106086708_15.zip
Uploading the intermediate file: upload_file_15.zip
  03:01:28 STEP 1536 H= 384:00 +CPU= 30.019
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001536
  03:01:57 STEP 1537 H= 384:15 +CPU= 28.325
  03:02:25 STEP 1538 H= 384:30 +CPU= 27.876
  03:02:56 STEP 1539 H= 384:45 +CPU= 29.861
  03:03:25 STEP 1540 H= 385:00 +CPU= 28.548
  03:03:54 STEP 1541 H= 385:15 +CPU= 28.254
  03:04:23 STEP 1542 H= 385:30 +CPU= 28.036
  03:04:50 STEP 1543 H= 385:45 +CPU= 27.452
  03:05:19 STEP 1544 H= 386:00 +CPU= 28.784
  03:05:49 STEP 1545 H= 386:15 +CPU= 28.915
  03:06:21 STEP 1546 H= 386:30 +CPU= 29.536
  03:06:48 STEP 1547 H= 386:45 +CPU= 27.326
  03:07:23 STEP 1548 H= 387:00 +CPU= 34.239
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001548
  03:07:51 STEP 1549 H= 387:15 +CPU= 28.151
  03:08:19 STEP 1550 H= 387:30 +CPU= 27.354
  03:08:46 STEP 1551 H= 387:45 +CPU= 27.120
  03:09:14 STEP 1552 H= 388:00 +CPU= 28.216
  03:09:42 STEP 1553 H= 388:15 +CPU= 27.555
  03:10:12 STEP 1554 H= 388:30 +CPU= 29.022
  03:10:40 STEP 1555 H= 388:45 +CPU= 28.461
  03:11:09 STEP 1556 H= 389:00 +CPU= 28.788
  03:11:37 STEP 1557 H= 389:15 +CPU= 27.424
  03:12:06 STEP 1558 H= 389:30 +CPU= 28.101
  03:12:34 STEP 1559 H= 389:45 +CPU= 27.993
  03:13:09 STEP 1560 H= 390:00 +CPU= 34.296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001560
  03:13:36 STEP 1561 H= 390:15 +CPU= 27.492
  03:14:05 STEP 1562 H= 390:30 +CPU= 28.113
  03:14:35 STEP 1563 H= 390:45 +CPU= 28.463
  03:15:04 STEP 1564 H= 391:00 +CPU= 28.925
  03:15:33 STEP 1565 H= 391:15 +CPU= 27.976
  03:16:03 STEP 1566 H= 391:30 +CPU= 28.781
  03:16:32 STEP 1567 H= 391:45 +CPU= 28.566
  03:17:02 STEP 1568 H= 392:00 +CPU= 29.004
  03:17:30 STEP 1569 H= 392:15 +CPU= 28.110
  03:17:59 STEP 1570 H= 392:30 +CPU= 28.261
  03:18:28 STEP 1571 H= 392:45 +CPU= 28.305
  03:19:05 STEP 1572 H= 393:00 +CPU= 35.624
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001572
  03:19:34 STEP 1573 H= 393:15 +CPU= 28.353
  03:20:02 STEP 1574 H= 393:30 +CPU= 28.206
  03:20:32 STEP 1575 H= 393:45 +CPU= 28.694
  03:21:02 STEP 1576 H= 394:00 +CPU= 29.043
  03:21:30 STEP 1577 H= 394:15 +CPU= 27.983
  03:21:59 STEP 1578 H= 394:30 +CPU= 28.407
  03:22:26 STEP 1579 H= 394:45 +CPU= 27.229
  03:22:57 STEP 1580 H= 395:00 +CPU= 29.564
  03:23:30 STEP 1581 H= 395:15 +CPU= 29.898
  03:24:12 STEP 1582 H= 395:30 +CPU= 34.654
  03:24:49 STEP 1583 H= 395:45 +CPU= 32.406
  03:25:34 STEP 1584 H= 396:00 +CPU= 41.131
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001584
  03:26:02 STEP 1585 H= 396:15 +CPU= 27.950
  03:26:31 STEP 1586 H= 396:30 +CPU= 27.895
  03:27:00 STEP 1587 H= 396:45 +CPU= 28.433
  03:27:30 STEP 1588 H= 397:00 +CPU= 29.428
  03:28:00 STEP 1589 H= 397:15 +CPU= 28.611
  03:28:28 STEP 1590 H= 397:30 +CPU= 27.803
  03:28:57 STEP 1591 H= 397:45 +CPU= 28.507
  03:29:26 STEP 1592 H= 398:00 +CPU= 28.712
  03:29:54 STEP 1593 H= 398:15 +CPU= 28.120
  03:30:22 STEP 1594 H= 398:30 +CPU= 27.592
  03:30:50 STEP 1595 H= 398:45 +CPU= 27.594
  03:31:26 STEP 1596 H= 399:00 +CPU= 35.613
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001596
  03:31:56 STEP 1597 H= 399:15 +CPU= 29.126
  03:32:25 STEP 1598 H= 399:30 +CPU= 28.089
  03:32:53 STEP 1599 H= 399:45 +CPU= 27.986
  03:33:23 STEP 1600 H= 400:00 +CPU= 29.194
  03:33:52 STEP 1601 H= 400:15 +CPU= 27.929
  03:34:21 STEP 1602 H= 400:30 +CPU= 28.427
  03:34:51 STEP 1603 H= 400:45 +CPU= 28.886
  03:35:20 STEP 1604 H= 401:00 +CPU= 28.788
  03:35:48 STEP 1605 H= 401:15 +CPU= 27.857
  03:36:18 STEP 1606 H= 401:30 +CPU= 28.722
  03:36:48 STEP 1607 H= 401:45 +CPU= 28.911
  03:37:30 STEP 1608 H= 402:00 +CPU= 38.737
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001608
  03:38:00 STEP 1609 H= 402:15 +CPU= 28.518
  03:38:28 STEP 1610 H= 402:30 +CPU= 27.869
  03:38:56 STEP 1611 H= 402:45 +CPU= 27.490
  03:39:26 STEP 1612 H= 403:00 +CPU= 29.552
  03:39:55 STEP 1613 H= 403:15 +CPU= 28.036
  03:40:23 STEP 1614 H= 403:30 +CPU= 27.348
  03:40:51 STEP 1615 H= 403:45 +CPU= 27.984
  03:41:27 STEP 1616 H= 404:00 +CPU= 31.994
  03:42:02 STEP 1617 H= 404:15 +CPU= 31.253
  03:42:35 STEP 1618 H= 404:30 +CPU= 30.944
  03:43:09 STEP 1619 H= 404:45 +CPU= 30.785
  03:43:49 STEP 1620 H= 405:00 +CPU= 38.210
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001620
  03:44:20 STEP 1621 H= 405:15 +CPU= 29.673
  03:44:49 STEP 1622 H= 405:30 +CPU= 27.935
  03:45:17 STEP 1623 H= 405:45 +CPU= 27.608
  03:45:45 STEP 1624 H= 406:00 +CPU= 28.253
  03:46:12 STEP 1625 H= 406:15 +CPU= 27.268
  03:46:40 STEP 1626 H= 406:30 +CPU= 27.200
  03:47:07 STEP 1627 H= 406:45 +CPU= 26.962
  03:47:34 STEP 1628 H= 407:00 +CPU= 27.507
  03:48:02 STEP 1629 H= 407:15 +CPU= 27.378
  03:48:29 STEP 1630 H= 407:30 +CPU= 27.054
  03:48:56 STEP 1631 H= 407:45 +CPU= 27.263
  03:49:34 STEP 1632 H= 408:00 +CPU= 37.696
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001632
  03:50:01 STEP 1633 H= 408:15 +CPU= 27.108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001632
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a199_2016092300_20_1018_12289956_0_r1106086708_16.zip
Uploading the intermediate file: upload_file_16.zip
  03:50:29 STEP 1634 H= 408:30 +CPU= 27.640
  03:50:57 STEP 1635 H= 408:45 +CPU= 27.320
  03:51:24 STEP 1636 H= 409:00 +CPU= 27.655
  03:51:52 STEP 1637 H= 409:15 +CPU= 27.253
  03:52:19 STEP 1638 H= 409:30 +CPU= 27.110
  03:52:47 STEP 1639 H= 409:45 +CPU= 27.922
  03:53:19 STEP 1640 H= 410:00 +CPU= 30.391
  03:53:51 STEP 1641 H= 410:15 +CPU= 29.805
  03:54:23 STEP 1642 H= 410:30 +CPU= 29.572
  03:54:55 STEP 1643 H= 410:45 +CPU= 29.965
  03:55:34 STEP 1644 H= 411:00 +CPU= 37.352
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001644
  03:56:02 STEP 1645 H= 411:15 +CPU= 27.853
  03:56:30 STEP 1646 H= 411:30 +CPU= 27.809
  03:56:58 STEP 1647 H= 411:45 +CPU= 27.534
  03:57:26 STEP 1648 H= 412:00 +CPU= 28.088
  03:57:54 STEP 1649 H= 412:15 +CPU= 27.705
  03:58:24 STEP 1650 H= 412:30 +CPU= 28.663
  03:58:51 STEP 1651 H= 412:45 +CPU= 27.422
  03:59:21 STEP 1652 H= 413:00 +CPU= 28.822
  03:59:49 STEP 1653 H= 413:15 +CPU= 27.581
  04:00:17 STEP 1654 H= 413:30 +CPU= 28.226
  04:00:46 STEP 1655 H= 413:45 +CPU= 28.275
  04:01:23 STEP 1656 H= 414:00 +CPU= 36.057
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001656
  04:01:52 STEP 1657 H= 414:15 +CPU= 28.304
  04:02:21 STEP 1658 H= 414:30 +CPU= 28.115
  04:02:50 STEP 1659 H= 414:45 +CPU= 28.224
  04:03:17 STEP 1660 H= 415:00 +CPU= 27.684
  04:03:45 STEP 1661 H= 415:15 +CPU= 27.623
  04:04:14 STEP 1662 H= 415:30 +CPU= 28.192
  04:04:43 STEP 1663 H= 415:45 +CPU= 28.185
  04:05:11 STEP 1664 H= 416:00 +CPU= 27.989
  04:05:40 STEP 1665 H= 416:15 +CPU= 28.241
  04:06:08 STEP 1666 H= 416:30 +CPU= 27.401
  04:06:36 STEP 1667 H= 416:45 +CPU= 27.857
  04:07:12 STEP 1668 H= 417:00 +CPU= 35.562
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001668
  04:07:42 STEP 1669 H= 417:15 +CPU= 28.529
  04:08:11 STEP 1670 H= 417:30 +CPU= 28.465
  04:08:41 STEP 1671 H= 417:45 +CPU= 28.470
  04:09:10 STEP 1672 H= 418:00 +CPU= 28.760
  04:09:39 STEP 1673 H= 418:15 +CPU= 28.410
  04:10:08 STEP 1674 H= 418:30 +CPU= 28.051
  04:10:36 STEP 1675 H= 418:45 +CPU= 28.086
  04:11:07 STEP 1676 H= 419:00 +CPU= 29.096
  04:11:35 STEP 1677 H= 419:15 +CPU= 27.977
  04:12:04 STEP 1678 H= 419:30 +CPU= 28.344
  04:12:32 STEP 1679 H= 419:45 +CPU= 28.006
  04:13:12 STEP 1680 H= 420:00 +CPU= 38.778
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001680
  04:13:41 STEP 1681 H= 420:15 +CPU= 28.106
  04:14:09 STEP 1682 H= 420:30 +CPU= 28.307
  04:14:37 STEP 1683 H= 420:45 +CPU= 27.734
  04:15:06 STEP 1684 H= 421:00 +CPU= 28.397
  04:15:36 STEP 1685 H= 421:15 +CPU= 28.585
  04:16:05 STEP 1686 H= 421:30 +CPU= 28.630
  04:16:34 STEP 1687 H= 421:45 +CPU= 28.320
  04:17:06 STEP 1688 H= 422:00 +CPU= 30.444
  04:17:36 STEP 1689 H= 422:15 +CPU= 28.953
  04:18:05 STEP 1690 H= 422:30 +CPU= 28.174
  04:18:33 STEP 1691 H= 422:45 +CPU= 28.049
  04:19:09 STEP 1692 H= 423:00 +CPU= 35.351
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001692
  04:19:38 STEP 1693 H= 423:15 +CPU= 28.097
  04:20:08 STEP 1694 H= 423:30 +CPU= 29.233
  04:20:38 STEP 1695 H= 423:45 +CPU= 28.892
  04:21:07 STEP 1696 H= 424:00 +CPU= 28.847
  04:21:36 STEP 1697 H= 424:15 +CPU= 28.623
  04:22:05 STEP 1698 H= 424:30 +CPU= 28.345
  04:22:33 STEP 1699 H= 424:45 +CPU= 27.618
  04:23:01 STEP 1700 H= 425:00 +CPU= 28.133
  04:23:29 STEP 1701 H= 425:15 +CPU= 27.752
  04:23:58 STEP 1702 H= 425:30 +CPU= 28.539
  04:24:27 STEP 1703 H= 425:45 +CPU= 28.320
  04:25:03 STEP 1704 H= 426:00 +CPU= 35.611
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001704
  04:25:32 STEP 1705 H= 426:15 +CPU= 27.937
  04:26:01 STEP 1706 H= 426:30 +CPU= 28.420
  04:26:32 STEP 1707 H= 426:45 +CPU= 29.433
  04:27:02 STEP 1708 H= 427:00 +CPU= 29.033
  04:27:32 STEP 1709 H= 427:15 +CPU= 29.473
  04:28:00 STEP 1710 H= 427:30 +CPU= 27.561
  04:28:27 STEP 1711 H= 427:45 +CPU= 27.255
  04:28:55 STEP 1712 H= 428:00 +CPU= 27.684
  04:29:22 STEP 1713 H= 428:15 +CPU= 27.129
  04:29:49 STEP 1714 H= 428:30 +CPU= 27.068
  04:30:16 STEP 1715 H= 428:45 +CPU= 26.613
  04:30:50 STEP 1716 H= 429:00 +CPU= 33.912
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001716
  04:31:17 STEP 1717 H= 429:15 +CPU= 27.064
  04:31:44 STEP 1718 H= 429:30 +CPU= 26.992
  04:32:11 STEP 1719 H= 429:45 +CPU= 26.981
  04:32:41 STEP 1720 H= 430:00 +CPU= 29.503
  04:33:10 STEP 1721 H= 430:15 +CPU= 28.411
  04:33:38 STEP 1722 H= 430:30 +CPU= 27.300
  04:34:05 STEP 1723 H= 430:45 +CPU= 27.340
  04:34:34 STEP 1724 H= 431:00 +CPU= 28.318
  04:35:02 STEP 1725 H= 431:15 +CPU= 27.576
  04:35:29 STEP 1726 H= 431:30 +CPU= 27.360
  04:35:57 STEP 1727 H= 431:45 +CPU= 27.023
  04:36:35 STEP 1728 H= 432:00 +CPU= 37.947
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001728
  04:37:04 STEP 1729 H= 432:15 +CPU= 28.823
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001728
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a199_2016092300_20_1018_12289956_0_r1106086708_17.zip
Uploading the intermediate file: upload_file_17.zip
  04:37:33 STEP 1730 H= 432:30 +CPU= 28.055
  04:38:01 STEP 1731 H= 432:45 +CPU= 27.648
  04:38:29 STEP 1732 H= 433:00 +CPU= 28.331
  04:38:57 STEP 1733 H= 433:15 +CPU= 27.447
  04:39:26 STEP 1734 H= 433:30 +CPU= 28.161
  04:39:56 STEP 1735 H= 433:45 +CPU= 28.827
  04:40:24 STEP 1736 H= 434:00 +CPU= 28.271
  04:40:53 STEP 1737 H= 434:15 +CPU= 27.972
  04:41:20 STEP 1738 H= 434:30 +CPU= 27.041
  04:41:47 STEP 1739 H= 434:45 +CPU= 26.787
  04:42:22 STEP 1740 H= 435:00 +CPU= 34.524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001740
  04:42:50 STEP 1741 H= 435:15 +CPU= 27.591
  04:43:20 STEP 1742 H= 435:30 +CPU= 28.891
  04:43:50 STEP 1743 H= 435:45 +CPU= 29.059
  04:44:22 STEP 1744 H= 436:00 +CPU= 30.554
  04:44:53 STEP 1745 H= 436:15 +CPU= 29.479
  04:45:24 STEP 1746 H= 436:30 +CPU= 29.461
  04:45:54 STEP 1747 H= 436:45 +CPU= 28.696
  04:46:23 STEP 1748 H= 437:00 +CPU= 28.697
  04:46:55 STEP 1749 H= 437:15 +CPU= 29.680
  04:47:26 STEP 1750 H= 437:30 +CPU= 29.798
  04:47:57 STEP 1751 H= 437:45 +CPU= 29.606
  04:48:33 STEP 1752 H= 438:00 +CPU= 35.046
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001752
  04:49:01 STEP 1753 H= 438:15 +CPU= 27.578
  04:49:30 STEP 1754 H= 438:30 +CPU= 28.142
  04:49:59 STEP 1755 H= 438:45 +CPU= 28.153
  04:50:28 STEP 1756 H= 439:00 +CPU= 28.579
  04:50:57 STEP 1757 H= 439:15 +CPU= 28.018
  04:51:26 STEP 1758 H= 439:30 +CPU= 28.532
  04:51:54 STEP 1759 H= 439:45 +CPU= 27.748
  04:52:24 STEP 1760 H= 440:00 +CPU= 28.667
  04:52:52 STEP 1761 H= 440:15 +CPU= 28.070
  04:53:22 STEP 1762 H= 440:30 +CPU= 28.221
  04:53:53 STEP 1763 H= 440:45 +CPU= 29.527
  04:54:28 STEP 1764 H= 441:00 +CPU= 34.567
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001764
  04:55:03 STEP 1765 H= 441:15 +CPU= 31.481
  04:55:41 STEP 1766 H= 441:30 +CPU= 32.879
  04:56:15 STEP 1767 H= 441:45 +CPU= 31.328
  04:56:48 STEP 1768 H= 442:00 +CPU= 31.451
  04:57:24 STEP 1769 H= 442:15 +CPU= 32.000
  04:57:59 STEP 1770 H= 442:30 +CPU= 32.299
  04:58:33 STEP 1771 H= 442:45 +CPU= 31.447
  04:59:05 STEP 1772 H= 443:00 +CPU= 29.940
  04:59:33 STEP 1773 H= 443:15 +CPU= 27.686
  05:00:02 STEP 1774 H= 443:30 +CPU= 28.383
  05:00:32 STEP 1775 H= 443:45 +CPU= 28.920
  05:01:13 STEP 1776 H= 444:00 +CPU= 40.010
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001776
  05:01:42 STEP 1777 H= 444:15 +CPU= 28.189
  05:02:11 STEP 1778 H= 444:30 +CPU= 28.444
  05:02:40 STEP 1779 H= 444:45 +CPU= 28.540
  05:03:10 STEP 1780 H= 445:00 +CPU= 29.460
  05:03:38 STEP 1781 H= 445:15 +CPU= 27.760
  05:04:08 STEP 1782 H= 445:30 +CPU= 28.750
  05:04:37 STEP 1783 H= 445:45 +CPU= 28.458
  05:05:06 STEP 1784 H= 446:00 +CPU= 28.215
  05:05:34 STEP 1785 H= 446:15 +CPU= 27.850
  05:06:02 STEP 1786 H= 446:30 +CPU= 27.718
  05:06:30 STEP 1787 H= 446:45 +CPU= 28.066
  05:07:06 STEP 1788 H= 447:00 +CPU= 35.028
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001788
  05:07:34 STEP 1789 H= 447:15 +CPU= 27.897
  05:08:02 STEP 1790 H= 447:30 +CPU= 27.978
  05:08:30 STEP 1791 H= 447:45 +CPU= 27.587
  05:08:59 STEP 1792 H= 448:00 +CPU= 28.651
  05:09:28 STEP 1793 H= 448:15 +CPU= 28.396
  05:09:57 STEP 1794 H= 448:30 +CPU= 28.085
  05:10:25 STEP 1795 H= 448:45 +CPU= 28.061
  05:10:54 STEP 1796 H= 449:00 +CPU= 28.409
  05:11:21 STEP 1797 H= 449:15 +CPU= 26.851
  05:11:48 STEP 1798 H= 449:30 +CPU= 26.768
  05:12:15 STEP 1799 H= 449:45 +CPU= 27.098
  05:12:48 STEP 1800 H= 450:00 +CPU= 33.583
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001800
  05:13:15 STEP 1801 H= 450:15 +CPU= 26.924
  05:13:42 STEP 1802 H= 450:30 +CPU= 26.716
  05:14:09 STEP 1803 H= 450:45 +CPU= 26.731
  05:14:36 STEP 1804 H= 451:00 +CPU= 27.330
  05:15:03 STEP 1805 H= 451:15 +CPU= 26.813
  05:15:30 STEP 1806 H= 451:30 +CPU= 26.926
  05:15:57 STEP 1807 H= 451:45 +CPU= 26.808
  05:16:24 STEP 1808 H= 452:00 +CPU= 27.441
  05:16:51 STEP 1809 H= 452:15 +CPU= 26.809
  05:17:18 STEP 1810 H= 452:30 +CPU= 26.604
  05:17:45 STEP 1811 H= 452:45 +CPU= 27.089
  05:18:19 STEP 1812 H= 453:00 +CPU= 33.598
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001812
  05:18:46 STEP 1813 H= 453:15 +CPU= 26.803
  05:19:13 STEP 1814 H= 453:30 +CPU= 26.971
  05:19:39 STEP 1815 H= 453:45 +CPU= 26.821
  05:20:07 STEP 1816 H= 454:00 +CPU= 27.522
  05:20:34 STEP 1817 H= 454:15 +CPU= 27.000
  05:21:01 STEP 1818 H= 454:30 +CPU= 27.003
  05:21:28 STEP 1819 H= 454:45 +CPU= 26.891
  05:21:56 STEP 1820 H= 455:00 +CPU= 27.413
  05:22:22 STEP 1821 H= 455:15 +CPU= 26.800
  05:22:49 STEP 1822 H= 455:30 +CPU= 26.709
  05:23:16 STEP 1823 H= 455:45 +CPU= 26.807
  05:23:53 STEP 1824 H= 456:00 +CPU= 37.386
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001824
  05:24:21 STEP 1825 H= 456:15 +CPU= 27.077
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001824
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a199_2016092300_20_1018_12289956_0_r1106086708_18.zip
Uploading the intermediate file: upload_file_18.zip
  05:24:48 STEP 1826 H= 456:30 +CPU= 27.271
  05:25:15 STEP 1827 H= 456:45 +CPU= 26.897
  05:25:42 STEP 1828 H= 457:00 +CPU= 27.346
  05:26:09 STEP 1829 H= 457:15 +CPU= 26.892
  05:26:36 STEP 1830 H= 457:30 +CPU= 26.962
  05:27:03 STEP 1831 H= 457:45 +CPU= 26.903
  05:27:31 STEP 1832 H= 458:00 +CPU= 27.563
  05:27:58 STEP 1833 H= 458:15 +CPU= 26.831
  05:28:24 STEP 1834 H= 458:30 +CPU= 26.763
  05:28:51 STEP 1835 H= 458:45 +CPU= 26.971
  05:29:25 STEP 1836 H= 459:00 +CPU= 33.617
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001836
  05:29:52 STEP 1837 H= 459:15 +CPU= 26.845
  05:30:19 STEP 1838 H= 459:30 +CPU= 26.917
  05:30:46 STEP 1839 H= 459:45 +CPU= 26.776
  05:31:13 STEP 1840 H= 460:00 +CPU= 27.441
  05:31:40 STEP 1841 H= 460:15 +CPU= 26.778
  05:32:07 STEP 1842 H= 460:30 +CPU= 26.873
  05:32:34 STEP 1843 H= 460:45 +CPU= 26.798
  05:33:01 STEP 1844 H= 461:00 +CPU= 27.286
  05:33:28 STEP 1845 H= 461:15 +CPU= 26.861
  05:33:55 STEP 1846 H= 461:30 +CPU= 26.783
  05:34:22 STEP 1847 H= 461:45 +CPU= 26.805
  05:34:56 STEP 1848 H= 462:00 +CPU= 33.690
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001848
  05:35:22 STEP 1849 H= 462:15 +CPU= 26.888
  05:35:49 STEP 1850 H= 462:30 +CPU= 26.860
  05:36:16 STEP 1851 H= 462:45 +CPU= 26.785
  05:36:44 STEP 1852 H= 463:00 +CPU= 27.416
  05:37:11 STEP 1853 H= 463:15 +CPU= 26.854
  05:37:38 STEP 1854 H= 463:30 +CPU= 26.901
  05:38:05 STEP 1855 H= 463:45 +CPU= 26.980
  05:38:32 STEP 1856 H= 464:00 +CPU= 27.523
  05:38:59 STEP 1857 H= 464:15 +CPU= 26.934
  05:39:26 STEP 1858 H= 464:30 +CPU= 26.981
  05:39:53 STEP 1859 H= 464:45 +CPU= 26.856
  05:40:27 STEP 1860 H= 465:00 +CPU= 33.603
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001860
  05:40:54 STEP 1861 H= 465:15 +CPU= 26.845
  05:41:20 STEP 1862 H= 465:30 +CPU= 26.767
  05:41:47 STEP 1863 H= 465:45 +CPU= 26.726
  05:42:15 STEP 1864 H= 466:00 +CPU= 27.424
  05:42:41 STEP 1865 H= 466:15 +CPU= 26.748
  05:43:08 STEP 1866 H= 466:30 +CPU= 26.838
  05:43:35 STEP 1867 H= 466:45 +CPU= 26.835
  05:44:03 STEP 1868 H= 467:00 +CPU= 27.330
  05:44:29 STEP 1869 H= 467:15 +CPU= 26.813
  05:44:56 STEP 1870 H= 467:30 +CPU= 26.931
  05:45:24 STEP 1871 H= 467:45 +CPU= 27.126
  05:46:01 STEP 1872 H= 468:00 +CPU= 37.480
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001872
  05:46:28 STEP 1873 H= 468:15 +CPU= 26.915
  05:46:55 STEP 1874 H= 468:30 +CPU= 26.978
  05:47:22 STEP 1875 H= 468:45 +CPU= 26.814
  05:47:50 STEP 1876 H= 469:00 +CPU= 27.644
  05:48:17 STEP 1877 H= 469:15 +CPU= 26.868
  05:48:44 STEP 1878 H= 469:30 +CPU= 26.817
  05:49:10 STEP 1879 H= 469:45 +CPU= 26.837
  05:49:38 STEP 1880 H= 470:00 +CPU= 27.404
  05:50:05 STEP 1881 H= 470:15 +CPU= 26.900
  05:50:32 STEP 1882 H= 470:30 +CPU= 26.937
  05:50:59 STEP 1883 H= 470:45 +CPU= 26.798
  05:51:32 STEP 1884 H= 471:00 +CPU= 33.513
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001884
  05:51:59 STEP 1885 H= 471:15 +CPU= 27.030
  05:52:26 STEP 1886 H= 471:30 +CPU= 26.812
  05:52:53 STEP 1887 H= 471:45 +CPU= 26.789
  05:53:20 STEP 1888 H= 472:00 +CPU= 27.246
  05:53:47 STEP 1889 H= 472:15 +CPU= 26.843
  05:54:14 STEP 1890 H= 472:30 +CPU= 26.836
  05:54:41 STEP 1891 H= 472:45 +CPU= 26.900
  05:55:08 STEP 1892 H= 473:00 +CPU= 27.391
  05:55:35 STEP 1893 H= 473:15 +CPU= 26.739
  05:56:02 STEP 1894 H= 473:30 +CPU= 26.931
  05:56:29 STEP 1895 H= 473:45 +CPU= 26.783
  05:57:03 STEP 1896 H= 474:00 +CPU= 33.512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001896
  05:57:29 STEP 1897 H= 474:15 +CPU= 26.895
  05:57:56 STEP 1898 H= 474:30 +CPU= 26.875
  05:58:23 STEP 1899 H= 474:45 +CPU= 26.779
  05:58:51 STEP 1900 H= 475:00 +CPU= 27.473
  05:59:18 STEP 1901 H= 475:15 +CPU= 26.875
  05:59:45 STEP 1902 H= 475:30 +CPU= 26.813
  06:00:12 STEP 1903 H= 475:45 +CPU= 26.979
  06:00:39 STEP 1904 H= 476:00 +CPU= 27.390
  06:01:06 STEP 1905 H= 476:15 +CPU= 26.703
  06:01:33 STEP 1906 H= 476:30 +CPU= 26.884
  06:02:00 STEP 1907 H= 476:45 +CPU= 26.784
  06:02:33 STEP 1908 H= 477:00 +CPU= 33.554
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001908
  06:03:00 STEP 1909 H= 477:15 +CPU= 26.970
  06:03:27 STEP 1910 H= 477:30 +CPU= 26.903
  06:03:54 STEP 1911 H= 477:45 +CPU= 26.837
  06:04:22 STEP 1912 H= 478:00 +CPU= 27.646
  06:04:48 STEP 1913 H= 478:15 +CPU= 26.770
  06:05:15 STEP 1914 H= 478:30 +CPU= 26.841
  06:05:42 STEP 1915 H= 478:45 +CPU= 26.909
  06:06:10 STEP 1916 H= 479:00 +CPU= 27.333
  06:06:37 STEP 1917 H= 479:15 +CPU= 26.786
  06:07:03 STEP 1918 H= 479:30 +CPU= 26.809
  06:07:30 STEP 1919 H= 479:45 +CPU= 26.803
  06:08:04 STEP 1920 H= 480:00 +CPU= 33.263
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.1383188362E-20      I       0.3271067009E-06      I
ICloud fractionI      0.0000000000E+00      I       0.1702258447E-01      I
I  Relat. Hum. I      0.2700000577E+00      I       0.3342533435E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1231317574E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    308650628 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS514417.7 Mbytes/s, TOTAL TIME=    0.0093( 6.5%)
 06:08:04 STEP 1920 H= 480:00 +CPU= 33.263
 END CNT3
 NSTEP =  1920 CNT0   000000000  24.001  24.001  24.055   0.998               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    25.8     3.9    21.9     4.7    43.9
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.94      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    4753.3    4753.3     17.65      0.00
   9 SPCM     - SPECTRAL COMP.                     384     978.5     978.5      3.35      0.00
  10 SCAN2M - PHYSICS                              385   14408.4   14408.4     49.38      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32     942.6     942.6      0.27      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      47.8      47.8      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      26.1      26.1      0.01      0.00
  27 RESTART FILES - WRITING                         8    3761.9    3761.9      0.27      0.00
  28 RESTART FILES - READING                         1    5635.6    5635.6      0.05      0.00
  29 SU4FPOS IN CNT4                                32       0.2       0.2      0.00      0.00
  30 DYNFPOS IN CNT4                                32    6361.0    6361.0      1.81      0.00
  31 POSDDH IN STEPO                                 2       7.0       7.0      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    5463.0    5463.0     18.72      0.00
  39 SU0YOMB                                         1     420.3     420.3      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      19.8      19.8      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     292.7     292.7      1.00      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      75.8      75.8      0.26      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     407.3     407.3      3.03      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     250.5     250.5      1.78      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800      82.7      82.7      0.59      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     130.9     130.9      0.97      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      52.0      52.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      59.5      59.5      0.44      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      33.3      33.3      0.24      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836      68.6      68.6      0.51      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      29.1      29.1      0.21      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    11232.8 CPU TIME   11023.5 VECTOR TIME    11023.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      153.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/4/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/4/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289956/ICMGGh7zg+001848
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a199_2016092300_20_1018_12289956_0_r1106086708_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
06:11:32 (2424): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Jun 2024 12:10:30 1549270 22443545 oifs_43r3_bl_a199_2016092300_20_1018_12289956_0 1,727,100 54,972 0.0318
14 Jun 2024 22:41:28 1549270 22443545 oifs_43r3_bl_a199_2016092300_20_1018_12289956_0 864,000 27,569 0.0319


©2024 cpdn.org