climateprediction.net home page
Task 22302569

Task 22302569

Name oifs_43r3_ps_0076_2019050100_123_988_12204720_1
Workunit 12204720
Created 23 Jan 2023, 22:37:03 UTC
Sent 23 Jan 2023, 22:46:43 UTC
Report deadline 24 Mar 2023, 22:46:43 UTC
Received 25 Jan 2023, 3:04:37 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1494567
Run time 15 hours 33 min 38 sec
CPU time 15 hours 28 min 54 sec
Validate state Invalid
Credit 7,620.48
Device peak FLOPS 4.54 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.26 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,238.16 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
rojects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_68.zip
Uploading the intermediate file: upload_file_68.zip
  14:52:40 STEP 1657 H=1657:00 +CPU= 22.538
  14:53:03 STEP 1658 H=1658:00 +CPU= 23.332
  14:53:39 STEP 1659 H=1659:00 +CPU= 35.816
  14:54:03 STEP 1660 H=1660:00 +CPU= 23.383
  14:54:26 STEP 1661 H=1661:00 +CPU= 22.780
  14:55:02 STEP 1662 H=1662:00 +CPU= 36.847
  14:55:25 STEP 1663 H=1663:00 +CPU= 22.816
  14:55:49 STEP 1664 H=1664:00 +CPU= 23.459
  14:56:25 STEP 1665 H=1665:00 +CPU= 36.048
  14:56:48 STEP 1666 H=1666:00 +CPU= 23.446
  14:57:11 STEP 1667 H=1667:00 +CPU= 22.807
  14:57:51 STEP 1668 H=1668:00 +CPU= 39.770
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001668
  14:58:14 STEP 1669 H=1669:00 +CPU= 22.646
  14:58:37 STEP 1670 H=1670:00 +CPU= 23.361
  14:59:13 STEP 1671 H=1671:00 +CPU= 35.875
  14:59:37 STEP 1672 H=1672:00 +CPU= 23.406
  14:59:59 STEP 1673 H=1673:00 +CPU= 22.698
  15:00:36 STEP 1674 H=1674:00 +CPU= 36.756
  15:00:59 STEP 1675 H=1675:00 +CPU= 22.691
  15:01:22 STEP 1676 H=1676:00 +CPU= 23.359
  15:01:58 STEP 1677 H=1677:00 +CPU= 35.959
  15:02:22 STEP 1678 H=1678:00 +CPU= 23.344
  15:02:45 STEP 1679 H=1679:00 +CPU= 23.320
  15:03:27 STEP 1680 H=1680:00 +CPU= 41.955
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_69.zip
Uploading the intermediate file: upload_file_69.zip
  15:03:50 STEP 1681 H=1681:00 +CPU= 22.507
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0076
(argv4) batchid: 988
(argv5) wuid: 12204720
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a9bc638f8f7b3360ae4604cbf7e586fe to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0076_2019050100_123_988_12204720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0076_2019050100_123_988_12204720.zip
ic_ancil_file: ic_ancil_12204720
ifsdata_file: ifsdata_12204720
climate_data_file: clim_data_12204720
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c8657570222c4482268b2b74ea26a4b2 to: /var/lib/boinc-client/slots/1/ic_ancil_12204720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12204720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6d4a6e5035f0d5920bb8207c09e7490 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12204720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12204720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12204720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12204720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12204720.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12204720.xml
-- Model is restarting --
Adjusting last_iter, 1681, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12204720.xml
last_cpu_time: 46501
upload_file_number: 70
last_iter: 1679
last_upload: 6048000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448
The child process has been launched with process id: 59226
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59226:59226] [20230124:151220:1674601940:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:13:14 STEP 1680 H=1680:00 +CPU= 40.772
  15:13:36 STEP 1681 H=1681:00 +CPU= 22.016
  15:13:59 STEP 1682 H=1682:00 +CPU= 22.848
  15:14:35 STEP 1683 H=1683:00 +CPU= 36.008
  15:14:59 STEP 1684 H=1684:00 +CPU= 23.412
  15:15:21 STEP 1685 H=1685:00 +CPU= 22.749
  15:15:58 STEP 1686 H=1686:00 +CPU= 36.757
  15:16:21 STEP 1687 H=1687:00 +CPU= 22.727
  15:16:45 STEP 1688 H=1688:00 +CPU= 23.362
  15:17:21 STEP 1689 H=1689:00 +CPU= 35.979
  15:17:44 STEP 1690 H=1690:00 +CPU= 23.347
  15:18:07 STEP 1691 H=1691:00 +CPU= 22.670
  15:18:47 STEP 1692 H=1692:00 +CPU= 39.647
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001692
  15:19:10 STEP 1693 H=1693:00 +CPU= 22.738
  15:19:33 STEP 1694 H=1694:00 +CPU= 23.354
  15:20:09 STEP 1695 H=1695:00 +CPU= 36.030
  15:20:32 STEP 1696 H=1696:00 +CPU= 23.384
  15:20:55 STEP 1697 H=1697:00 +CPU= 22.687
  15:21:32 STEP 1698 H=1698:00 +CPU= 36.742
  15:21:55 STEP 1699 H=1699:00 +CPU= 22.661
  15:22:18 STEP 1700 H=1700:00 +CPU= 23.304
  15:22:54 STEP 1701 H=1701:00 +CPU= 35.955
  15:23:17 STEP 1702 H=1702:00 +CPU= 23.314
  15:23:41 STEP 1703 H=1703:00 +CPU= 23.314
  15:24:23 STEP 1704 H=1704:00 +CPU= 41.787
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_70.zip
Uploading the intermediate file: upload_file_70.zip
  15:24:46 STEP 1705 H=1705:00 +CPU= 22.681
  15:25:10 STEP 1706 H=1706:00 +CPU= 23.341
  15:25:46 STEP 1707 H=1707:00 +CPU= 36.059
  15:26:09 STEP 1708 H=1708:00 +CPU= 23.443
  15:26:32 STEP 1709 H=1709:00 +CPU= 22.748
  15:27:09 STEP 1710 H=1710:00 +CPU= 36.867
  15:27:32 STEP 1711 H=1711:00 +CPU= 22.761
  15:27:55 STEP 1712 H=1712:00 +CPU= 23.371
  15:28:31 STEP 1713 H=1713:00 +CPU= 36.080
  15:28:55 STEP 1714 H=1714:00 +CPU= 23.266
  15:29:19 STEP 1715 H=1715:00 +CPU= 22.793
  15:29:58 STEP 1716 H=1716:00 +CPU= 38.571
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001716
  15:30:20 STEP 1717 H=1717:00 +CPU= 22.608
  15:30:44 STEP 1718 H=1718:00 +CPU= 23.372
  15:31:20 STEP 1719 H=1719:00 +CPU= 36.066
  15:31:43 STEP 1720 H=1720:00 +CPU= 23.403
  15:32:06 STEP 1721 H=1721:00 +CPU= 22.707
  15:32:43 STEP 1722 H=1722:00 +CPU= 36.846
  15:33:06 STEP 1723 H=1723:00 +CPU= 22.756
  15:33:29 STEP 1724 H=1724:00 +CPU= 23.391
  15:34:05 STEP 1725 H=1725:00 +CPU= 36.004
  15:34:29 STEP 1726 H=1726:00 +CPU= 23.322
  15:34:52 STEP 1727 H=1727:00 +CPU= 23.308
  15:35:34 STEP 1728 H=1728:00 +CPU= 41.800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_71.zip
Uploading the intermediate file: upload_file_71.zip
  15:35:57 STEP 1729 H=1729:00 +CPU= 22.714
  15:36:20 STEP 1730 H=1730:00 +CPU= 23.340
  15:36:56 STEP 1731 H=1731:00 +CPU= 36.105
  15:37:20 STEP 1732 H=1732:00 +CPU= 23.471
  15:37:43 STEP 1733 H=1733:00 +CPU= 22.783
  15:38:20 STEP 1734 H=1734:00 +CPU= 36.792
  15:38:43 STEP 1735 H=1735:00 +CPU= 22.725
  15:39:06 STEP 1736 H=1736:00 +CPU= 23.387
  15:39:42 STEP 1737 H=1737:00 +CPU= 36.039
  15:40:05 STEP 1738 H=1738:00 +CPU= 23.385
  15:40:28 STEP 1739 H=1739:00 +CPU= 22.712
  15:41:08 STEP 1740 H=1740:00 +CPU= 39.722
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001740
  15:41:31 STEP 1741 H=1741:00 +CPU= 22.744
  15:41:54 STEP 1742 H=1742:00 +CPU= 23.366
  15:42:30 STEP 1743 H=1743:00 +CPU= 36.043
  15:42:54 STEP 1744 H=1744:00 +CPU= 23.403
  15:43:16 STEP 1745 H=1745:00 +CPU= 22.693
  15:43:53 STEP 1746 H=1746:00 +CPU= 36.744
  15:44:16 STEP 1747 H=1747:00 +CPU= 22.672
  15:44:39 STEP 1748 H=1748:00 +CPU= 23.318
  15:45:15 STEP 1749 H=1749:00 +CPU= 35.970
  15:45:39 STEP 1750 H=1750:00 +CPU= 23.311
  15:46:02 STEP 1751 H=1751:00 +CPU= 23.346
  15:46:44 STEP 1752 H=1752:00 +CPU= 41.795
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_72.zip
Uploading the intermediate file: upload_file_72.zip
  15:47:07 STEP 1753 H=1753:00 +CPU= 22.726
  15:47:30 STEP 1754 H=1754:00 +CPU= 23.324
  15:48:07 STEP 1755 H=1755:00 +CPU= 36.102
  15:48:30 STEP 1756 H=1756:00 +CPU= 23.429
  15:48:53 STEP 1757 H=1757:00 +CPU= 22.826
  15:49:30 STEP 1758 H=1758:00 +CPU= 36.827
  15:49:53 STEP 1759 H=1759:00 +CPU= 22.725
  15:50:16 STEP 1760 H=1760:00 +CPU= 23.392
  15:50:52 STEP 1761 H=1761:00 +CPU= 36.075
  15:51:16 STEP 1762 H=1762:00 +CPU= 23.354
  15:51:38 STEP 1763 H=1763:00 +CPU= 22.739
  15:52:18 STEP 1764 H=1764:00 +CPU= 39.712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001764
  15:52:41 STEP 1765 H=1765:00 +CPU= 22.781
  15:53:04 STEP 1766 H=1766:00 +CPU= 23.417
  15:53:41 STEP 1767 H=1767:00 +CPU= 36.046
  15:54:04 STEP 1768 H=1768:00 +CPU= 23.400
  15:54:27 STEP 1769 H=1769:00 +CPU= 22.783
  15:55:04 STEP 1770 H=1770:00 +CPU= 36.786
  15:55:26 STEP 1771 H=1771:00 +CPU= 22.712
  15:55:50 STEP 1772 H=1772:00 +CPU= 23.400
  15:56:26 STEP 1773 H=1773:00 +CPU= 36.015
  15:56:49 STEP 1774 H=1774:00 +CPU= 23.381
  15:57:13 STEP 1775 H=1775:00 +CPU= 23.375
  15:57:55 STEP 1776 H=1776:00 +CPU= 41.774
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_73.zip
Uploading the intermediate file: upload_file_73.zip
  15:58:17 STEP 1777 H=1777:00 +CPU= 22.702
  15:58:41 STEP 1778 H=1778:00 +CPU= 23.335
  15:59:17 STEP 1779 H=1779:00 +CPU= 36.061
  15:59:40 STEP 1780 H=1780:00 +CPU= 23.392
  16:00:03 STEP 1781 H=1781:00 +CPU= 22.782
  16:00:40 STEP 1782 H=1782:00 +CPU= 36.833
  16:01:03 STEP 1783 H=1783:00 +CPU= 22.731
  16:01:26 STEP 1784 H=1784:00 +CPU= 23.407
  16:02:03 STEP 1785 H=1785:00 +CPU= 36.074
  16:02:26 STEP 1786 H=1786:00 +CPU= 23.404
  16:02:49 STEP 1787 H=1787:00 +CPU= 22.740
  16:03:29 STEP 1788 H=1788:00 +CPU= 39.726
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001788
  16:03:51 STEP 1789 H=1789:00 +CPU= 22.764
  16:04:15 STEP 1790 H=1790:00 +CPU= 23.451
  16:04:51 STEP 1791 H=1791:00 +CPU= 36.067
  16:05:14 STEP 1792 H=1792:00 +CPU= 23.425
  16:05:37 STEP 1793 H=1793:00 +CPU= 22.874
  16:06:16 STEP 1794 H=1794:00 +CPU= 36.845
  16:06:38 STEP 1795 H=1795:00 +CPU= 22.042
  16:07:01 STEP 1796 H=1796:00 +CPU= 23.006
  16:07:37 STEP 1797 H=1797:00 +CPU= 35.939
  16:08:00 STEP 1798 H=1798:00 +CPU= 23.401
  16:08:24 STEP 1799 H=1799:00 +CPU= 23.509
  16:09:06 STEP 1800 H=1800:00 +CPU= 41.785
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_74.zip
Uploading the intermediate file: upload_file_74.zip
  16:09:29 STEP 1801 H=1801:00 +CPU= 22.595
  16:09:52 STEP 1802 H=1802:00 +CPU= 23.234
  16:10:28 STEP 1803 H=1803:00 +CPU= 36.025
  16:10:52 STEP 1804 H=1804:00 +CPU= 23.297
  16:11:14 STEP 1805 H=1805:00 +CPU= 22.616
  16:11:51 STEP 1806 H=1806:00 +CPU= 36.832
  16:12:14 STEP 1807 H=1807:00 +CPU= 22.704
  16:12:37 STEP 1808 H=1808:00 +CPU= 23.383
  16:13:14 STEP 1809 H=1809:00 +CPU= 36.075
  16:13:37 STEP 1810 H=1810:00 +CPU= 23.389
  16:14:00 STEP 1811 H=1811:00 +CPU= 22.735
  16:14:40 STEP 1812 H=1812:00 +CPU= 39.819
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001812
  16:15:02 STEP 1813 H=1813:00 +CPU= 22.726
  16:15:26 STEP 1814 H=1814:00 +CPU= 23.378
  16:16:02 STEP 1815 H=1815:00 +CPU= 36.052
  16:16:25 STEP 1816 H=1816:00 +CPU= 23.335
  16:16:48 STEP 1817 H=1817:00 +CPU= 22.639
  16:17:25 STEP 1818 H=1818:00 +CPU= 36.756
  16:17:48 STEP 1819 H=1819:00 +CPU= 22.706
  16:18:11 STEP 1820 H=1820:00 +CPU= 23.408
  16:18:47 STEP 1821 H=1821:00 +CPU= 35.946
  16:19:11 STEP 1822 H=1822:00 +CPU= 23.428
  16:19:34 STEP 1823 H=1823:00 +CPU= 23.499
  16:20:16 STEP 1824 H=1824:00 +CPU= 41.920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_75.zip
Uploading the intermediate file: upload_file_75.zip
  16:20:39 STEP 1825 H=1825:00 +CPU= 22.623
  16:21:03 STEP 1826 H=1826:00 +CPU= 23.281
  16:21:39 STEP 1827 H=1827:00 +CPU= 36.033
  16:22:02 STEP 1828 H=1828:00 +CPU= 23.130
  16:23:12 STEP 1829 H=1829:00 +CPU= 24.413
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0076
(argv4) batchid: 988
(argv5) wuid: 12204720
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a9bc638f8f7b3360ae4604cbf7e586fe to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0076_2019050100_123_988_12204720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0076_2019050100_123_988_12204720.zip
ic_ancil_file: ic_ancil_12204720
ifsdata_file: ifsdata_12204720
climate_data_file: clim_data_12204720
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c8657570222c4482268b2b74ea26a4b2 to: /var/lib/boinc-client/slots/1/ic_ancil_12204720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12204720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6d4a6e5035f0d5920bb8207c09e7490 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12204720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12204720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12204720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12204720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12204720.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12204720.xml
-- Model is restarting --
Adjusting last_iter, 1830, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12204720.xml
last_cpu_time: 50684
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448
The child process has been launched with process id: 59429
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59429:59429] [20230124:163031:1674606631:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:31:52 STEP 1824 H=1824:00 +CPU= 41.699
  16:32:14 STEP 1825 H=1825:00 +CPU= 22.078
  16:32:37 STEP 1826 H=1826:00 +CPU= 22.705
  16:33:13 STEP 1827 H=1827:00 +CPU= 35.800
  16:33:36 STEP 1828 H=1828:00 +CPU= 23.440
  16:33:59 STEP 1829 H=1829:00 +CPU= 22.768
  16:34:36 STEP 1830 H=1830:00 +CPU= 36.849
  16:34:59 STEP 1831 H=1831:00 +CPU= 22.683
  16:35:23 STEP 1832 H=1832:00 +CPU= 23.385
  16:35:59 STEP 1833 H=1833:00 +CPU= 36.039
  16:36:22 STEP 1834 H=1834:00 +CPU= 23.349
  16:36:45 STEP 1835 H=1835:00 +CPU= 22.711
  16:37:25 STEP 1836 H=1836:00 +CPU= 39.675
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001836
  16:37:47 STEP 1837 H=1837:00 +CPU= 22.711
  16:38:11 STEP 1838 H=1838:00 +CPU= 23.423
  16:38:47 STEP 1839 H=1839:00 +CPU= 35.959
  16:39:10 STEP 1840 H=1840:00 +CPU= 23.395
  16:39:33 STEP 1841 H=1841:00 +CPU= 22.684
  16:40:10 STEP 1842 H=1842:00 +CPU= 36.893
  16:40:33 STEP 1843 H=1843:00 +CPU= 22.722
  16:40:56 STEP 1844 H=1844:00 +CPU= 23.387
  16:41:32 STEP 1845 H=1845:00 +CPU= 36.080
  16:41:56 STEP 1846 H=1846:00 +CPU= 23.393
  16:42:19 STEP 1847 H=1847:00 +CPU= 23.342
  16:43:01 STEP 1848 H=1848:00 +CPU= 41.628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_76.zip
Uploading the intermediate file: upload_file_76.zip
  16:43:24 STEP 1849 H=1849:00 +CPU= 22.680
  16:43:48 STEP 1850 H=1850:00 +CPU= 23.388
  16:44:24 STEP 1851 H=1851:00 +CPU= 35.955
  16:44:47 STEP 1852 H=1852:00 +CPU= 23.380
  16:45:10 STEP 1853 H=1853:00 +CPU= 22.705
  16:45:47 STEP 1854 H=1854:00 +CPU= 36.946
  16:46:10 STEP 1855 H=1855:00 +CPU= 22.736
  16:46:34 STEP 1856 H=1856:00 +CPU= 23.421
  16:47:10 STEP 1857 H=1857:00 +CPU= 36.107
  16:47:33 STEP 1858 H=1858:00 +CPU= 23.335
  16:47:57 STEP 1859 H=1859:00 +CPU= 23.340
  16:48:36 STEP 1860 H=1860:00 +CPU= 39.812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001860
  16:48:58 STEP 1861 H=1861:00 +CPU= 22.034
  16:49:21 STEP 1862 H=1862:00 +CPU= 22.930
  16:49:58 STEP 1863 H=1863:00 +CPU= 36.090
  16:50:21 STEP 1864 H=1864:00 +CPU= 23.421
  16:50:44 STEP 1865 H=1865:00 +CPU= 22.812
  16:51:21 STEP 1866 H=1866:00 +CPU= 36.879
  16:51:43 STEP 1867 H=1867:00 +CPU= 22.669
  16:52:07 STEP 1868 H=1868:00 +CPU= 23.315
  16:52:43 STEP 1869 H=1869:00 +CPU= 36.040
  16:53:06 STEP 1870 H=1870:00 +CPU= 23.269
  16:53:30 STEP 1871 H=1871:00 +CPU= 23.345
  16:54:12 STEP 1872 H=1872:00 +CPU= 41.696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_77.zip
Uploading the intermediate file: upload_file_77.zip
  16:54:35 STEP 1873 H=1873:00 +CPU= 22.634
  16:54:58 STEP 1874 H=1874:00 +CPU= 23.356
  16:55:34 STEP 1875 H=1875:00 +CPU= 36.012
  16:55:57 STEP 1876 H=1876:00 +CPU= 23.359
  16:56:20 STEP 1877 H=1877:00 +CPU= 22.665
  16:56:57 STEP 1878 H=1878:00 +CPU= 36.947
  16:57:20 STEP 1879 H=1879:00 +CPU= 22.574
  16:57:43 STEP 1880 H=1880:00 +CPU= 23.415
  16:58:19 STEP 1881 H=1881:00 +CPU= 36.004
  16:58:43 STEP 1882 H=1882:00 +CPU= 23.282
  16:59:05 STEP 1883 H=1883:00 +CPU= 22.664
  16:59:45 STEP 1884 H=1884:00 +CPU= 39.876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001884
  17:00:08 STEP 1885 H=1885:00 +CPU= 22.782
  17:00:32 STEP 1886 H=1886:00 +CPU= 23.462
  17:01:08 STEP 1887 H=1887:00 +CPU= 36.158
  17:01:31 STEP 1888 H=1888:00 +CPU= 23.438
  17:01:54 STEP 1889 H=1889:00 +CPU= 22.757
  17:02:31 STEP 1890 H=1890:00 +CPU= 36.936
  17:02:54 STEP 1891 H=1891:00 +CPU= 22.582
  17:03:17 STEP 1892 H=1892:00 +CPU= 23.403
  17:03:53 STEP 1893 H=1893:00 +CPU= 36.094
  17:04:17 STEP 1894 H=1894:00 +CPU= 23.268
  17:04:40 STEP 1895 H=1895:00 +CPU= 23.408
  17:05:22 STEP 1896 H=1896:00 +CPU= 41.906
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_78.zip
Uploading the intermediate file: upload_file_78.zip
  17:05:45 STEP 1897 H=1897:00 +CPU= 22.664
  17:06:09 STEP 1898 H=1898:00 +CPU= 23.323
  17:06:45 STEP 1899 H=1899:00 +CPU= 36.026
  17:07:08 STEP 1900 H=1900:00 +CPU= 23.383
  17:07:31 STEP 1901 H=1901:00 +CPU= 22.686
  17:08:08 STEP 1902 H=1902:00 +CPU= 36.864
  17:08:31 STEP 1903 H=1903:00 +CPU= 22.617
  17:08:54 STEP 1904 H=1904:00 +CPU= 23.564
  17:09:31 STEP 1905 H=1905:00 +CPU= 36.318
  17:09:54 STEP 1906 H=1906:00 +CPU= 23.334
  17:10:17 STEP 1907 H=1907:00 +CPU= 22.854
  17:10:57 STEP 1908 H=1908:00 +CPU= 39.747
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001908
  17:11:20 STEP 1909 H=1909:00 +CPU= 22.665
  17:11:43 STEP 1910 H=1910:00 +CPU= 23.345
  17:12:19 STEP 1911 H=1911:00 +CPU= 36.045
  17:12:42 STEP 1912 H=1912:00 +CPU= 23.305
  17:13:05 STEP 1913 H=1913:00 +CPU= 22.618
  17:13:42 STEP 1914 H=1914:00 +CPU= 36.878
  17:14:05 STEP 1915 H=1915:00 +CPU= 22.647
  17:14:28 STEP 1916 H=1916:00 +CPU= 23.503
  17:15:05 STEP 1917 H=1917:00 +CPU= 36.331
  17:15:28 STEP 1918 H=1918:00 +CPU= 23.271
  17:15:51 STEP 1919 H=1919:00 +CPU= 23.543
  17:16:34 STEP 1920 H=1920:00 +CPU= 42.015
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_79.zip
Uploading the intermediate file: upload_file_79.zip
  17:16:56 STEP 1921 H=1921:00 +CPU= 22.660
  17:17:20 STEP 1922 H=1922:00 +CPU= 23.390
  17:17:56 STEP 1923 H=1923:00 +CPU= 36.087
  17:18:19 STEP 1924 H=1924:00 +CPU= 23.397
  17:18:42 STEP 1925 H=1925:00 +CPU= 22.708
  17:19:19 STEP 1926 H=1926:00 +CPU= 36.973
  17:19:42 STEP 1927 H=1927:00 +CPU= 22.572
  17:20:05 STEP 1928 H=1928:00 +CPU= 23.457
  17:20:42 STEP 1929 H=1929:00 +CPU= 36.381
  17:21:05 STEP 1930 H=1930:00 +CPU= 23.307
  17:21:28 STEP 1931 H=1931:00 +CPU= 22.852
  17:22:08 STEP 1932 H=1932:00 +CPU= 39.789
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001932
  17:22:31 STEP 1933 H=1933:00 +CPU= 22.617
  17:22:54 STEP 1934 H=1934:00 +CPU= 23.391
  17:23:30 STEP 1935 H=1935:00 +CPU= 35.998
  17:23:54 STEP 1936 H=1936:00 +CPU= 23.266
  17:24:16 STEP 1937 H=1937:00 +CPU= 22.576
  17:24:53 STEP 1938 H=1938:00 +CPU= 36.829
  17:25:16 STEP 1939 H=1939:00 +CPU= 22.862
  17:25:38 STEP 1940 H=1940:00 +CPU= 22.443
  17:26:14 STEP 1941 H=1941:00 +CPU= 35.138
  17:26:37 STEP 1942 H=1942:00 +CPU= 23.332
  17:27:00 STEP 1943 H=1943:00 +CPU= 23.020
  17:27:43 STEP 1944 H=1944:00 +CPU= 42.142
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_80.zip
Uploading the intermediate file: upload_file_80.zip
  17:28:06 STEP 1945 H=1945:00 +CPU= 22.662
  17:28:29 STEP 1946 H=1946:00 +CPU= 23.176
  17:29:05 STEP 1947 H=1947:00 +CPU= 36.304
  17:29:29 STEP 1948 H=1948:00 +CPU= 23.342
  17:29:51 STEP 1949 H=1949:00 +CPU= 22.562
  17:30:28 STEP 1950 H=1950:00 +CPU= 37.077
  17:30:51 STEP 1951 H=1951:00 +CPU= 22.766
  17:31:14 STEP 1952 H=1952:00 +CPU= 23.133
  17:31:51 STEP 1953 H=1953:00 +CPU= 36.422
  17:32:14 STEP 1954 H=1954:00 +CPU= 23.401
  17:32:37 STEP 1955 H=1955:00 +CPU= 22.475
  17:33:17 STEP 1956 H=1956:00 +CPU= 40.159
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001956
  17:33:39 STEP 1957 H=1957:00 +CPU= 22.698
  17:34:03 STEP 1958 H=1958:00 +CPU= 23.227
  17:34:39 STEP 1959 H=1959:00 +CPU= 36.274
  17:35:02 STEP 1960 H=1960:00 +CPU= 23.320
  17:35:25 STEP 1961 H=1961:00 +CPU= 22.513
  17:36:02 STEP 1962 H=1962:00 +CPU= 37.004
  17:36:25 STEP 1963 H=1963:00 +CPU= 22.750
  17:36:48 STEP 1964 H=1964:00 +CPU= 23.037
  17:37:24 STEP 1965 H=1965:00 +CPU= 36.260
  17:37:48 STEP 1966 H=1966:00 +CPU= 23.321
  17:38:11 STEP 1967 H=1967:00 +CPU= 22.969
  17:38:53 STEP 1968 H=1968:00 +CPU= 42.184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_81.zip
Uploading the intermediate file: upload_file_81.zip
  17:39:16 STEP 1969 H=1969:00 +CPU= 22.629
  17:39:39 STEP 1970 H=1970:00 +CPU= 23.163
  17:40:16 STEP 1971 H=1971:00 +CPU= 36.211
  17:40:39 STEP 1972 H=1972:00 +CPU= 23.317
  17:41:01 STEP 1973 H=1973:00 +CPU= 22.417
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0076
(argv4) batchid: 988
(argv5) wuid: 12204720
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a9bc638f8f7b3360ae4604cbf7e586fe to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0076_2019050100_123_988_12204720.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0076_2019050100_123_988_12204720.zip
ic_ancil_file: ic_ancil_12204720
ifsdata_file: ifsdata_12204720
climate_data_file: clim_data_12204720
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c8657570222c4482268b2b74ea26a4b2 to: /var/lib/boinc-client/slots/1/ic_ancil_12204720.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12204720.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6d4a6e5035f0d5920bb8207c09e7490 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12204720.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12204720.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12204720.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12204720.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12204720.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12204720.xml
-- Model is restarting --
Adjusting last_iter, 1974, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12204720.xml
last_cpu_time: 54881
upload_file_number: 82
last_iter: 1967
last_upload: 7084800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448
The child process has been launched with process id: 59702
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-Boinc-01:1:1:59702:59702] [20230124:174935:1674611375:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:50:29 STEP 1968 H=1968:00 +CPU= 40.094
  17:50:51 STEP 1969 H=1969:00 +CPU= 21.996
  17:51:15 STEP 1970 H=1970:00 +CPU= 23.182
  17:52:16 STEP 1971 H=1971:00 +CPU= 36.137
  17:52:39 STEP 1972 H=1972:00 +CPU= 22.609
  17:53:01 STEP 1973 H=1973:00 +CPU= 21.978
  17:53:37 STEP 1974 H=1974:00 +CPU= 36.137
  17:54:00 STEP 1975 H=1975:00 +CPU= 22.719
  17:54:23 STEP 1976 H=1976:00 +CPU= 23.395
  17:54:59 STEP 1977 H=1977:00 +CPU= 35.939
  17:55:23 STEP 1978 H=1978:00 +CPU= 23.408
  17:55:46 STEP 1979 H=1979:00 +CPU= 22.727
  17:56:25 STEP 1980 H=1980:00 +CPU= 39.661
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001980
  17:56:48 STEP 1981 H=1981:00 +CPU= 22.809
  17:57:12 STEP 1982 H=1982:00 +CPU= 23.502
  17:57:48 STEP 1983 H=1983:00 +CPU= 36.011
  17:58:11 STEP 1984 H=1984:00 +CPU= 23.481
  17:58:34 STEP 1985 H=1985:00 +CPU= 22.785
  17:59:11 STEP 1986 H=1986:00 +CPU= 36.772
  17:59:34 STEP 1987 H=1987:00 +CPU= 22.707
  17:59:57 STEP 1988 H=1988:00 +CPU= 23.373
  18:00:33 STEP 1989 H=1989:00 +CPU= 35.951
  18:00:57 STEP 1990 H=1990:00 +CPU= 23.383
  18:01:20 STEP 1991 H=1991:00 +CPU= 23.313
  18:02:02 STEP 1992 H=1992:00 +CPU= 41.738
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204720/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0076_2019050100_123_988_12204720_1_r1202964448_82.zip
Uploading the intermediate file: upload_file_82.zip
  18:02:25 STEP 1993 H=1993:00 +CPU= 22.649
  18:02:48 STEP 1994 H=1994:00 +CPU= 23.351
  18:03:24 STEP 1995 H=1995:00 +CPU= 35.990
  18:03:48 STEP 1996 H=1996:00 +CPU= 23.368
  18:04:11 STEP 1997 H=1997:00 +CPU= 22.720
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:39:53 1494567 22302569 oifs_43r3_ps_0076_2019050100_123_988_12204720_1 5,443,200 41,309 0.0076
09 May 2023 16:39:45 1494567 22302569 oifs_43r3_ps_0076_2019050100_123_988_12204720_1 4,579,200 34,838 0.0076
09 May 2023 16:39:42 1494567 22302569 oifs_43r3_ps_0076_2019050100_123_988_12204720_1 3,715,200 28,357 0.0076
09 May 2023 16:39:35 1494567 22302569 oifs_43r3_ps_0076_2019050100_123_988_12204720_1 2,851,200 21,862 0.0077
09 May 2023 16:39:24 1494567 22302569 oifs_43r3_ps_0076_2019050100_123_988_12204720_1 1,987,200 14,910 0.0075
09 May 2023 16:39:12 1494567 22302569 oifs_43r3_ps_0076_2019050100_123_988_12204720_1 864,000 6,589 0.0076


©2024 cpdn.org