climateprediction.net home page
Task 22442154

Task 22442154

Name oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0
Workunit 12288598
Created 12 Jun 2024, 20:22:09 UTC
Sent 12 Jun 2024, 20:54:24 UTC
Report deadline 11 Aug 2024, 20:54:24 UTC
Received 11 Jul 2024, 19:50:11 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1541083
Run time 2 days 3 hours 39 min 25 sec
CPU time 2 days 3 hours 39 min 25 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 2.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,011.66 MB
Peak swap size 5,981.07 MB
Peak disk usage 2,231.38 MB
Stderr
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
U= 84.517
  05:56:06 STEP 1578 H= 394:30 +CPU= 97.703
  05:57:41 STEP 1579 H= 394:45 +CPU= 94.721
  05:58:57 STEP 1580 H= 395:00 +CPU= 76.052
  06:00:16 STEP 1581 H= 395:15 +CPU= 78.006
  06:02:05 STEP 1582 H= 395:30 +CPU=107.974
  06:03:39 STEP 1583 H= 395:45 +CPU= 92.597
  06:05:37 STEP 1584 H= 396:00 +CPU=113.999
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001584
  06:07:18 STEP 1585 H= 396:15 +CPU= 99.369
  06:08:59 STEP 1586 H= 396:30 +CPU= 99.894
  06:10:16 STEP 1587 H= 396:45 +CPU= 76.987
  06:11:31 STEP 1588 H= 397:00 +CPU= 74.046
  06:13:21 STEP 1589 H= 397:15 +CPU=105.968
  06:15:04 STEP 1590 H= 397:30 +CPU=100.400
  06:16:29 STEP 1591 H= 397:45 +CPU= 84.287
  06:17:43 STEP 1592 H= 398:00 +CPU= 74.050
  06:19:25 STEP 1593 H= 398:15 +CPU= 99.874
  06:21:06 STEP 1594 H= 398:30 +CPU= 98.628
  06:22:26 STEP 1595 H= 398:45 +CPU= 79.470
  06:23:56 STEP 1596 H= 399:00 +CPU= 89.579
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001596
  06:25:54 STEP 1597 H= 399:15 +CPU=114.210
  06:27:30 STEP 1598 H= 399:30 +CPU= 94.988
  06:28:49 STEP 1599 H= 399:45 +CPU= 77.308
  06:30:09 STEP 1600 H= 400:00 +CPU= 78.870
  06:31:45 STEP 1601 H= 400:15 +CPU= 93.741
  06:33:19 STEP 1602 H= 400:30 +CPU= 92.837
  06:34:36 STEP 1603 H= 400:45 +CPU= 76.563
  06:35:53 STEP 1604 H= 401:00 +CPU= 75.587
  06:37:33 STEP 1605 H= 401:15 +CPU= 98.289
  06:39:03 STEP 1606 H= 401:30 +CPU= 89.217
  06:40:21 STEP 1607 H= 401:45 +CPU= 77.480
  06:42:18 STEP 1608 H= 402:00 +CPU=114.261
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001608
  06:44:30 STEP 1609 H= 402:15 +CPU=130.526
  06:46:21 STEP 1610 H= 402:30 +CPU=109.560
  06:48:07 STEP 1611 H= 402:45 +CPU=104.150
  06:50:02 STEP 1612 H= 403:00 +CPU=113.405
  06:51:37 STEP 1613 H= 403:15 +CPU= 92.959
  06:53:18 STEP 1614 H= 403:30 +CPU= 99.988
  06:55:07 STEP 1615 H= 403:45 +CPU=107.274
  06:57:07 STEP 1616 H= 404:00 +CPU=119.255
  06:58:43 STEP 1617 H= 404:15 +CPU= 94.621
  07:00:17 STEP 1618 H= 404:30 +CPU= 94.090
  07:02:05 STEP 1619 H= 404:45 +CPU=106.152
  07:04:31 STEP 1620 H= 405:00 +CPU=144.278
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001620
  07:06:08 STEP 1621 H= 405:15 +CPU= 96.511
  07:08:13 STEP 1622 H= 405:30 +CPU=121.935
  07:10:01 STEP 1623 H= 405:45 +CPU=107.468
  07:11:57 STEP 1624 H= 406:00 +CPU=114.297
  07:13:30 STEP 1625 H= 406:15 +CPU= 92.608
  07:15:39 STEP 1626 H= 406:30 +CPU=127.489
  07:17:10 STEP 1627 H= 406:45 +CPU= 90.223
  07:18:37 STEP 1628 H= 407:00 +CPU= 85.345
  07:19:55 STEP 1629 H= 407:15 +CPU= 76.855
  07:21:30 STEP 1630 H= 407:30 +CPU= 91.971
  07:23:09 STEP 1631 H= 407:45 +CPU= 98.470
  07:25:06 STEP 1632 H= 408:00 +CPU=115.394
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0_r2106100285_16.zip
Uploading the intermediate file: upload_file_16.zip
  07:26:30 STEP 1633 H= 408:15 +CPU= 82.524
  07:28:04 STEP 1634 H= 408:30 +CPU= 91.285
  07:29:39 STEP 1635 H= 408:45 +CPU= 93.771
  07:31:14 STEP 1636 H= 409:00 +CPU= 93.058
  07:32:28 STEP 1637 H= 409:15 +CPU= 72.848
  07:34:01 STEP 1638 H= 409:30 +CPU= 91.520
  07:35:42 STEP 1639 H= 409:45 +CPU=100.390
  07:37:20 STEP 1640 H= 410:00 +CPU= 96.999
  07:38:36 STEP 1641 H= 410:15 +CPU= 75.807
  07:39:55 STEP 1642 H= 410:30 +CPU= 77.993
  07:41:24 STEP 1643 H= 410:45 +CPU= 87.143
  07:43:26 STEP 1644 H= 411:00 +CPU=120.482
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001644
  07:44:41 STEP 1645 H= 411:15 +CPU= 74.178
  07:45:55 STEP 1646 H= 411:30 +CPU= 74.310
  07:47:17 STEP 1647 H= 411:45 +CPU= 80.610
  07:49:01 STEP 1648 H= 412:00 +CPU=102.556
  07:50:27 STEP 1649 H= 412:15 +CPU= 85.249
  07:51:47 STEP 1650 H= 412:30 +CPU= 79.401
  07:53:01 STEP 1651 H= 412:45 +CPU= 73.240
  07:54:27 STEP 1652 H= 413:00 +CPU= 82.675
  07:56:17 STEP 1653 H= 413:15 +CPU=107.990
  07:57:35 STEP 1654 H= 413:30 +CPU= 77.381
  07:58:57 STEP 1655 H= 413:45 +CPU= 80.793
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a07j
(argv4) batchid: 1018
(argv5) wuid: 12288598
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d2b6b4f2cf8f22ac09cb68e3c33464e4 to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a07j_2016092300_20_1018_12288598.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a07j_2016092300_20_1018_12288598.zip
ic_ancil_file: ic_ancil_12288598
ifsdata_file: ifsdata_12288598
climate_data_file: clim_data_12288598
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/9/ic_ancil_12288598.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12288598.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2ea26e4fd553a810727bba9b853cca41 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12288598.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12288598.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12288598.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12288598.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12288598.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12288598.xml
-- Model is restarting --
Adjusting last_iter, 1656, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12288598.xml
last_cpu_time: 159511
upload_file_number: 17
last_iter: 1633
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0_r2106100285
The child process has been launched with process id: 1958
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:1958:1958] [20240711:223045:1720693845:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:33:22 STEP 1632 H= 408:00 +CPU=137.188
  22:35:39 STEP 1633 H= 408:15 +CPU=123.563
  22:37:52 STEP 1634 H= 408:30 +CPU=109.120
  22:40:04 STEP 1635 H= 408:45 +CPU=127.230
  22:42:12 STEP 1636 H= 409:00 +CPU=110.764
  22:44:11 STEP 1637 H= 409:15 +CPU=113.415
  22:45:37 STEP 1638 H= 409:30 +CPU= 84.593
  22:47:13 STEP 1639 H= 409:45 +CPU= 96.037
  22:48:32 STEP 1640 H= 410:00 +CPU= 78.308
  22:49:49 STEP 1641 H= 410:15 +CPU= 76.375
  22:51:22 STEP 1642 H= 410:30 +CPU= 92.159
  22:52:38 STEP 1643 H= 410:45 +CPU= 75.609
  22:54:01 STEP 1644 H= 411:00 +CPU= 82.936
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001644
  22:55:36 STEP 1645 H= 411:15 +CPU= 92.747
  22:56:49 STEP 1646 H= 411:30 +CPU= 72.928
  22:57:58 STEP 1647 H= 411:45 +CPU= 69.321
  22:58:56 STEP 1648 H= 412:00 +CPU= 57.063
  23:00:14 STEP 1649 H= 412:15 +CPU= 78.521
  23:01:46 STEP 1650 H= 412:30 +CPU= 90.935
  23:03:10 STEP 1651 H= 412:45 +CPU= 83.403
  23:04:27 STEP 1652 H= 413:00 +CPU= 76.428
  23:05:58 STEP 1653 H= 413:15 +CPU= 89.065
  23:07:41 STEP 1654 H= 413:30 +CPU=100.992
  23:09:01 STEP 1655 H= 413:45 +CPU= 79.214
  23:10:30 STEP 1656 H= 414:00 +CPU= 88.153
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001656
  23:12:03 STEP 1657 H= 414:15 +CPU= 91.910
  23:13:23 STEP 1658 H= 414:30 +CPU= 79.309
  23:14:46 STEP 1659 H= 414:45 +CPU= 82.328
  23:16:01 STEP 1660 H= 415:00 +CPU= 74.237
  23:17:37 STEP 1661 H= 415:15 +CPU= 94.677
  23:19:19 STEP 1662 H= 415:30 +CPU=101.297
  23:21:09 STEP 1663 H= 415:45 +CPU=108.416
  23:22:55 STEP 1664 H= 416:00 +CPU=104.448
  23:25:08 STEP 1665 H= 416:15 +CPU=131.300
  23:26:54 STEP 1666 H= 416:30 +CPU=104.887
  23:28:40 STEP 1667 H= 416:45 +CPU=103.986
  23:31:14 STEP 1668 H= 417:00 +CPU=151.891
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001668
  23:32:46 STEP 1669 H= 417:15 +CPU= 91.590
  23:34:21 STEP 1670 H= 417:30 +CPU= 93.442
  23:36:48 STEP 1671 H= 417:45 +CPU=141.775
  23:38:18 STEP 1672 H= 418:00 +CPU= 85.408
  23:40:00 STEP 1673 H= 418:15 +CPU=100.545
  23:41:49 STEP 1674 H= 418:30 +CPU=106.065
  23:43:11 STEP 1675 H= 418:45 +CPU= 80.539
  23:44:33 STEP 1676 H= 419:00 +CPU= 80.630
  23:46:15 STEP 1677 H= 419:15 +CPU=100.981
  23:48:05 STEP 1678 H= 419:30 +CPU=108.711
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:49:52 STEP 1679 H= 419:45 +CPU= 85.022
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:34 STEP 1680 H= 420:00 +CPU=116.894
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001680
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:44 STEP 1681 H= 420:15 +CPU=108.406
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:57:13 STEP 1682 H= 420:30 +CPU=103.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:59:27 STEP 1683 H= 420:45 +CPU= 89.704
  00:00:54 STEP 1684 H= 421:00 +CPU= 84.362
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:02:54 STEP 1685 H= 421:15 +CPU= 98.385
  00:04:24 STEP 1686 H= 421:30 +CPU= 87.037
  00:05:52 STEP 1687 H= 421:45 +CPU= 86.993
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:07:33 STEP 1688 H= 422:00 +CPU= 79.984
  00:09:32 STEP 1689 H= 422:15 +CPU=117.146
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:11:24 STEP 1690 H= 422:30 +CPU= 89.580
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:13:30 STEP 1691 H= 422:45 +CPU= 83.493
  00:15:13 STEP 1692 H= 423:00 +CPU=102.276
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001692
  00:17:00 STEP 1693 H= 423:15 +CPU=106.242
  00:18:44 STEP 1694 H= 423:30 +CPU=101.360
  00:20:06 STEP 1695 H= 423:45 +CPU= 81.007
  00:21:08 STEP 1696 H= 424:00 +CPU= 61.954
  00:22:28 STEP 1697 H= 424:15 +CPU= 79.295
  00:24:12 STEP 1698 H= 424:30 +CPU=100.740
  00:25:34 STEP 1699 H= 424:45 +CPU= 81.646
  00:27:09 STEP 1700 H= 425:00 +CPU= 92.497
  00:28:19 STEP 1701 H= 425:15 +CPU= 69.987
  00:29:44 STEP 1702 H= 425:30 +CPU= 84.229
  00:31:16 STEP 1703 H= 425:45 +CPU= 91.152
  00:33:11 STEP 1704 H= 426:00 +CPU=112.626
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001704
  00:34:41 STEP 1705 H= 426:15 +CPU= 89.645
  00:35:47 STEP 1706 H= 426:30 +CPU= 65.780
  00:37:08 STEP 1707 H= 426:45 +CPU= 80.263
  00:38:44 STEP 1708 H= 427:00 +CPU= 94.707
  00:40:12 STEP 1709 H= 427:15 +CPU= 87.809
  00:41:46 STEP 1710 H= 427:30 +CPU= 92.159
  00:43:14 STEP 1711 H= 427:45 +CPU= 87.661
  00:44:48 STEP 1712 H= 428:00 +CPU= 92.879
  00:46:28 STEP 1713 H= 428:15 +CPU= 99.519
  00:47:58 STEP 1714 H= 428:30 +CPU= 87.223
  00:49:08 STEP 1715 H= 428:45 +CPU= 70.668
  00:50:52 STEP 1716 H= 429:00 +CPU=102.196
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001716
  00:52:34 STEP 1717 H= 429:15 +CPU=101.821
  00:54:09 STEP 1718 H= 429:30 +CPU= 93.660
  00:55:41 STEP 1719 H= 429:45 +CPU= 90.973
  00:57:05 STEP 1720 H= 430:00 +CPU= 83.249
  00:58:31 STEP 1721 H= 430:15 +CPU= 85.401
  01:00:16 STEP 1722 H= 430:30 +CPU=104.067
  01:01:53 STEP 1723 H= 430:45 +CPU= 95.038
  01:03:20 STEP 1724 H= 431:00 +CPU= 85.771
  01:04:23 STEP 1725 H= 431:15 +CPU= 63.039
  01:05:45 STEP 1726 H= 431:30 +CPU= 80.964
  01:07:26 STEP 1727 H= 431:45 +CPU=100.103
  01:09:48 STEP 1728 H= 432:00 +CPU=139.040
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0_r2106100285_17.zip
Uploading the intermediate file: upload_file_17.zip
  01:11:07 STEP 1729 H= 432:15 +CPU= 76.882
  01:12:22 STEP 1730 H= 432:30 +CPU= 74.477
  01:14:15 STEP 1731 H= 432:45 +CPU=107.540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:16:11 STEP 1732 H= 433:00 +CPU= 93.066
  01:17:38 STEP 1733 H= 433:15 +CPU= 85.014
  01:18:52 STEP 1734 H= 433:30 +CPU= 73.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:20:26 STEP 1735 H= 433:45 +CPU= 71.495
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:22:39 STEP 1736 H= 434:00 +CPU=110.204
  01:23:50 STEP 1737 H= 434:15 +CPU= 71.026
  01:25:24 STEP 1738 H= 434:30 +CPU= 92.848
  01:26:46 STEP 1739 H= 434:45 +CPU= 80.680
  01:28:32 STEP 1740 H= 435:00 +CPU=103.298
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001740
  01:30:04 STEP 1741 H= 435:15 +CPU= 91.457
  01:31:23 STEP 1742 H= 435:30 +CPU= 78.148
  01:32:40 STEP 1743 H= 435:45 +CPU= 76.239
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:34:23 STEP 1744 H= 436:00 +CPU= 81.632
  01:36:10 STEP 1745 H= 436:15 +CPU=105.718
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:38:29 STEP 1746 H= 436:30 +CPU=117.182
  01:40:08 STEP 1747 H= 436:45 +CPU= 96.461
  01:41:20 STEP 1748 H= 437:00 +CPU= 71.411
  01:42:45 STEP 1749 H= 437:15 +CPU= 84.743
  01:44:35 STEP 1750 H= 437:30 +CPU=108.585
  01:46:12 STEP 1751 H= 437:45 +CPU= 96.733
  01:48:07 STEP 1752 H= 438:00 +CPU=113.584
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001752
  01:49:32 STEP 1753 H= 438:15 +CPU= 83.672
  01:51:19 STEP 1754 H= 438:30 +CPU=106.414
  01:52:59 STEP 1755 H= 438:45 +CPU= 97.908
  01:54:26 STEP 1756 H= 439:00 +CPU= 86.411
  01:55:51 STEP 1757 H= 439:15 +CPU= 83.807
  01:57:24 STEP 1758 H= 439:30 +CPU= 91.639
  01:59:14 STEP 1759 H= 439:45 +CPU=108.656
  02:00:52 STEP 1760 H= 440:00 +CPU= 97.231
  02:02:24 STEP 1761 H= 440:15 +CPU= 91.304
  02:04:10 STEP 1762 H= 440:30 +CPU=105.621
  02:05:45 STEP 1763 H= 440:45 +CPU= 94.129
  02:07:45 STEP 1764 H= 441:00 +CPU=117.663
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001764
  02:09:03 STEP 1765 H= 441:15 +CPU= 76.864
  02:10:26 STEP 1766 H= 441:30 +CPU= 81.579
  02:12:02 STEP 1767 H= 441:45 +CPU= 94.692
  02:13:26 STEP 1768 H= 442:00 +CPU= 83.543
  02:14:51 STEP 1769 H= 442:15 +CPU= 83.328
  02:16:10 STEP 1770 H= 442:30 +CPU= 78.561
  02:17:32 STEP 1771 H= 442:45 +CPU= 81.301
  02:19:15 STEP 1772 H= 443:00 +CPU=101.651
  02:20:38 STEP 1773 H= 443:15 +CPU= 82.113
  02:21:45 STEP 1774 H= 443:30 +CPU= 67.317
  02:22:48 STEP 1775 H= 443:45 +CPU= 62.901
  02:24:59 STEP 1776 H= 444:00 +CPU=129.354
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001776
  02:26:22 STEP 1777 H= 444:15 +CPU= 81.114
  02:28:14 STEP 1778 H= 444:30 +CPU=110.247
  02:29:38 STEP 1779 H= 444:45 +CPU= 82.079
  02:30:54 STEP 1780 H= 445:00 +CPU= 75.448
  02:32:51 STEP 1781 H= 445:15 +CPU=116.167
  02:34:20 STEP 1782 H= 445:30 +CPU= 87.625
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a07j
(argv4) batchid: 1018
(argv5) wuid: 12288598
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d2b6b4f2cf8f22ac09cb68e3c33464e4 to: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a07j_2016092300_20_1018_12288598.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_bl_a07j_2016092300_20_1018_12288598.zip
ic_ancil_file: ic_ancil_12288598
ifsdata_file: ifsdata_12288598
climate_data_file: clim_data_12288598
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/9/ic_ancil_12288598.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12288598.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2ea26e4fd553a810727bba9b853cca41 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12288598.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12288598.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12288598.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12288598.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12288598.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12288598.xml
-- Model is restarting --
Adjusting last_iter, 1783, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12288598.xml
last_cpu_time: 173519
upload_file_number: 18
last_iter: 1777
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0_r2106100285
The child process has been launched with process id: 3063
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.003] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.004] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.004] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.004] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:hpc1:1:1:3063:3063] [20240712:023945:1720708785:0.004] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:41:59 STEP 1776 H= 444:00 +CPU=114.337
  02:43:11 STEP 1777 H= 444:15 +CPU= 70.884
  02:44:23 STEP 1778 H= 444:30 +CPU= 71.604
  02:45:32 STEP 1779 H= 444:45 +CPU= 69.305
  02:46:49 STEP 1780 H= 445:00 +CPU= 76.200
  02:48:27 STEP 1781 H= 445:15 +CPU= 97.590
  02:49:49 STEP 1782 H= 445:30 +CPU= 81.376
  02:50:59 STEP 1783 H= 445:45 +CPU= 69.385
  02:52:11 STEP 1784 H= 446:00 +CPU= 71.989
  02:53:45 STEP 1785 H= 446:15 +CPU= 93.911
  02:55:13 STEP 1786 H= 446:30 +CPU= 86.808
  02:56:48 STEP 1787 H= 446:45 +CPU= 94.553
  02:58:21 STEP 1788 H= 447:00 +CPU= 92.596
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001788
  02:59:29 STEP 1789 H= 447:15 +CPU= 66.522
  03:01:13 STEP 1790 H= 447:30 +CPU=103.225
  03:02:33 STEP 1791 H= 447:45 +CPU= 79.221
  03:04:00 STEP 1792 H= 448:00 +CPU= 86.379
  03:05:23 STEP 1793 H= 448:15 +CPU= 82.219
  03:06:38 STEP 1794 H= 448:30 +CPU= 74.050
  03:08:31 STEP 1795 H= 448:45 +CPU=112.853
  03:09:51 STEP 1796 H= 449:00 +CPU= 78.951
  03:11:02 STEP 1797 H= 449:15 +CPU= 70.934
  03:12:11 STEP 1798 H= 449:30 +CPU= 69.128
  03:13:44 STEP 1799 H= 449:45 +CPU= 91.351
  03:15:34 STEP 1800 H= 450:00 +CPU=109.108
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001800
  03:16:46 STEP 1801 H= 450:15 +CPU= 71.278
  03:18:06 STEP 1802 H= 450:30 +CPU= 79.222
  03:19:15 STEP 1803 H= 450:45 +CPU= 68.615
  03:20:37 STEP 1804 H= 451:00 +CPU= 80.958
  03:22:20 STEP 1805 H= 451:15 +CPU=101.644
  03:23:44 STEP 1806 H= 451:30 +CPU= 83.638
  03:25:01 STEP 1807 H= 451:45 +CPU= 76.525
  03:26:09 STEP 1808 H= 452:00 +CPU= 67.509
  03:27:12 STEP 1809 H= 452:15 +CPU= 63.134
  03:29:06 STEP 1810 H= 452:30 +CPU=112.000
  03:30:30 STEP 1811 H= 452:45 +CPU= 82.875
  03:32:08 STEP 1812 H= 453:00 +CPU= 97.307
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001812
  03:33:15 STEP 1813 H= 453:15 +CPU= 66.426
  03:34:29 STEP 1814 H= 453:30 +CPU= 74.163
  03:36:01 STEP 1815 H= 453:45 +CPU= 90.619
  03:37:23 STEP 1816 H= 454:00 +CPU= 81.169
  03:38:35 STEP 1817 H= 454:15 +CPU= 71.030
  03:39:41 STEP 1818 H= 454:30 +CPU= 65.325
  03:41:11 STEP 1819 H= 454:45 +CPU= 88.221
  03:43:04 STEP 1820 H= 455:00 +CPU=111.077
  03:44:17 STEP 1821 H= 455:15 +CPU= 72.515
  03:46:03 STEP 1822 H= 455:30 +CPU=103.328
  03:47:23 STEP 1823 H= 455:45 +CPU= 78.948
  03:49:20 STEP 1824 H= 456:00 +CPU=115.331
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0_r2106100285_18.zip
Uploading the intermediate file: upload_file_18.zip
  03:51:11 STEP 1825 H= 456:15 +CPU=108.852
  03:52:31 STEP 1826 H= 456:30 +CPU= 79.430
  03:53:57 STEP 1827 H= 456:45 +CPU= 85.294
  03:55:20 STEP 1828 H= 457:00 +CPU= 82.174
  03:56:54 STEP 1829 H= 457:15 +CPU= 91.693
  03:58:30 STEP 1830 H= 457:30 +CPU= 95.115
  04:00:04 STEP 1831 H= 457:45 +CPU= 92.511
  04:01:23 STEP 1832 H= 458:00 +CPU= 77.858
  04:02:49 STEP 1833 H= 458:15 +CPU= 84.071
  04:04:18 STEP 1834 H= 458:30 +CPU= 88.688
  04:05:40 STEP 1835 H= 458:45 +CPU= 81.499
  04:07:38 STEP 1836 H= 459:00 +CPU=116.027
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001836
  04:08:41 STEP 1837 H= 459:15 +CPU= 63.362
  04:10:07 STEP 1838 H= 459:30 +CPU= 84.619
  04:11:51 STEP 1839 H= 459:45 +CPU=102.652
  04:13:07 STEP 1840 H= 460:00 +CPU= 75.906
  04:14:26 STEP 1841 H= 460:15 +CPU= 77.746
  04:15:48 STEP 1842 H= 460:30 +CPU= 80.988
  04:17:04 STEP 1843 H= 460:45 +CPU= 75.226
  04:18:41 STEP 1844 H= 461:00 +CPU= 96.207
  04:20:05 STEP 1845 H= 461:15 +CPU= 83.325
  04:21:23 STEP 1846 H= 461:30 +CPU= 76.702
  04:22:39 STEP 1847 H= 461:45 +CPU= 75.317
  04:23:51 STEP 1848 H= 462:00 +CPU= 71.845
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001848
  04:25:07 STEP 1849 H= 462:15 +CPU= 75.348
  04:26:52 STEP 1850 H= 462:30 +CPU=103.555
  04:28:04 STEP 1851 H= 462:45 +CPU= 71.383
  04:29:13 STEP 1852 H= 463:00 +CPU= 67.825
  04:30:20 STEP 1853 H= 463:15 +CPU= 66.783
  04:31:40 STEP 1854 H= 463:30 +CPU= 79.430
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:34:17 STEP 1855 H= 463:45 +CPU= 92.445
  04:35:49 STEP 1856 H= 464:00 +CPU= 92.211
  04:36:58 STEP 1857 H= 464:15 +CPU= 67.988
  04:38:19 STEP 1858 H= 464:30 +CPU= 80.369
  04:39:31 STEP 1859 H= 464:45 +CPU= 71.096
  04:40:57 STEP 1860 H= 465:00 +CPU= 85.024
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001860
  04:42:42 STEP 1861 H= 465:15 +CPU=104.904
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:44:22 STEP 1862 H= 465:30 +CPU= 78.033
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:46:34 STEP 1863 H= 465:45 +CPU= 88.353
  04:47:47 STEP 1864 H= 466:00 +CPU= 73.149
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:49:15 STEP 1865 H= 466:15 +CPU= 66.499
  04:50:59 STEP 1866 H= 466:30 +CPU=103.215
  04:52:26 STEP 1867 H= 466:45 +CPU= 86.472
  04:53:56 STEP 1868 H= 467:00 +CPU= 88.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:56:22 STEP 1869 H= 467:15 +CPU= 82.137
  04:57:45 STEP 1870 H= 467:30 +CPU= 82.186
  04:59:15 STEP 1871 H= 467:45 +CPU= 88.986
  05:01:30 STEP 1872 H= 468:00 +CPU=134.750
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001872
  05:02:49 STEP 1873 H= 468:15 +CPU= 78.083
  05:04:09 STEP 1874 H= 468:30 +CPU= 79.682
  05:05:31 STEP 1875 H= 468:45 +CPU= 80.540
  05:07:00 STEP 1876 H= 469:00 +CPU= 88.436
  05:08:22 STEP 1877 H= 469:15 +CPU= 81.014
  05:09:38 STEP 1878 H= 469:30 +CPU= 75.691
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:49:13 STEP 1879 H= 469:45 +CPU= 70.721
  05:50:38 STEP 1880 H= 470:00 +CPU= 82.822
  05:51:59 STEP 1881 H= 470:15 +CPU= 79.696
  05:53:03 STEP 1882 H= 470:30 +CPU= 64.321
  05:54:39 STEP 1883 H= 470:45 +CPU= 93.065
  05:56:45 STEP 1884 H= 471:00 +CPU=124.874
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:58:47 STEP 1885 H= 471:15 +CPU= 99.136
  05:59:53 STEP 1886 H= 471:30 +CPU= 65.682
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:01:26 STEP 1887 H= 471:45 +CPU= 71.898
  06:02:58 STEP 1888 H= 472:00 +CPU= 89.883
  06:04:49 STEP 1889 H= 472:15 +CPU=108.915
  06:06:25 STEP 1890 H= 472:30 +CPU= 94.678
  06:07:49 STEP 1891 H= 472:45 +CPU= 83.789
  06:09:05 STEP 1892 H= 473:00 +CPU= 75.587
  06:10:53 STEP 1893 H= 473:15 +CPU=105.935
  06:12:37 STEP 1894 H= 473:30 +CPU=101.076
  06:14:08 STEP 1895 H= 473:45 +CPU= 88.752
  06:15:38 STEP 1896 H= 474:00 +CPU= 89.933
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001896
  06:16:49 STEP 1897 H= 474:15 +CPU= 71.204
  06:18:44 STEP 1898 H= 474:30 +CPU=112.247
  06:20:23 STEP 1899 H= 474:45 +CPU= 97.864
  06:21:47 STEP 1900 H= 475:00 +CPU= 82.808
  06:23:05 STEP 1901 H= 475:15 +CPU= 78.023
  06:24:13 STEP 1902 H= 475:30 +CPU= 67.448
  06:26:02 STEP 1903 H= 475:45 +CPU=107.861
  06:27:39 STEP 1904 H= 476:00 +CPU= 95.349
  06:29:16 STEP 1905 H= 476:15 +CPU= 95.818
  06:30:29 STEP 1906 H= 476:30 +CPU= 72.224
  06:31:41 STEP 1907 H= 476:45 +CPU= 71.658
  06:33:59 STEP 1908 H= 477:00 +CPU=135.532
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001908
  06:35:37 STEP 1909 H= 477:15 +CPU= 96.085
  06:37:00 STEP 1910 H= 477:30 +CPU= 82.520
  06:38:12 STEP 1911 H= 477:45 +CPU= 71.342
  06:39:51 STEP 1912 H= 478:00 +CPU= 96.682
  06:41:36 STEP 1913 H= 478:15 +CPU=104.194
  06:43:04 STEP 1914 H= 478:30 +CPU= 85.962
  06:44:28 STEP 1915 H= 478:45 +CPU= 83.749
  06:45:32 STEP 1916 H= 479:00 +CPU= 63.864
  06:46:42 STEP 1917 H= 479:15 +CPU= 69.548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:48:58 STEP 1918 H= 479:30 +CPU=110.402
  06:50:25 STEP 1919 H= 479:45 +CPU= 86.354
  06:52:27 STEP 1920 H= 480:00 +CPU=119.493
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.4929087202E-21      I       0.4524634110E-07      I
ICloud fractionI      0.0000000000E+00      I       0.2128533532E-01      I
I  Relat. Hum. I      0.1260668868E+00      I       0.2055769678E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1061994924E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    299942564 BYTES IN      7 RECORDS TRANSFERRED IN    0.0023 SECONDS130409.8 Mbytes/s, TOTAL TIME=    0.0243( 9.5%)
 06:52:27 STEP 1920 H= 480:00 +CPU=119.493
 END CNT3
 NSTEP =  1920 CNT0   000000000  82.730  82.730  83.473   0.991               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         4290/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     78    0       46      1044068     4290 ************    36.5     8.0    28.4    13.8    49.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.88      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  157   13943.6   13943.6     14.44      0.00
   9 SPCM     - SPECTRAL COMP.                     144    2001.0    2001.0      1.90      0.00
  10 SCAN2M - PHYSICS                              145   55858.6   55858.6     53.42      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 12    3227.4    3227.4      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  7     124.4     124.4      0.01      0.00
  17 GRIDFPOS IN CNT4                               12      95.5      95.5      0.01      0.00
  27 RESTART FILES - WRITING                         3   12907.6   12907.6      0.26      0.00
  28 RESTART FILES - READING                         1   16867.8   16867.8      0.11      0.00
  29 SU4FPOS IN CNT4                                12       1.3       1.3      0.00      0.00
  30 DYNFPOS IN CNT4                                12   17894.3   17894.3      1.42      0.00
  31 POSDDH IN STEPO                                 1       5.7       5.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    145   19953.4   19953.4     19.08      0.00
  39 SU0YOMB                                         1    1487.0    1487.0      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    145     642.3     642.3      0.61      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              144     860.4     860.4      0.82      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              144     254.0     254.0      0.24      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  7       2.3       2.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      316    1314.2    1314.2      2.74      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       300     668.0     668.0      1.32      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        300     286.5     286.5      0.57      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       316     357.0     357.0      0.74      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1     121.7     121.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            316     353.2     353.2      0.74      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            300     160.7     160.7      0.32      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            316     255.3     255.3      0.53      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            300     186.4     186.4      0.37      0.00
 400 GSTATS                                      17763       0.1       0.1      0.01      0.00
 401 GSTATS HOOK                                 16473       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    15161.5 CPU TIME   12446.3 VECTOR TIME    12446.3


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      114.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/9/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/9/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12288598/ICMSHh7zg+001920
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0_r2106100285_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
06:56:06 (3060): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Jul 2024 19:51:56 1541083 22442154 oifs_43r3_bl_a07j_2016092300_20_1018_12288598_0 1,727,100 185,965 0.1077


©2024 cpdn.org