climateprediction.net home page
Task 22270260

Task 22270260

Name oifs_43r3_ps_0333_1999050100_123_968_12184977_0
Workunit 12184977
Created 22 Dec 2022, 18:33:02 UTC
Sent 19 Jan 2023, 0:52:35 UTC
Report deadline 20 Mar 2023, 0:52:35 UTC
Received 5 Feb 2023, 15:43:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1509518
Run time 11 hours 16 min 16 sec
CPU time 9 hours 51 min 10 sec
Validate state Invalid
Credit 3,507.84
Device peak FLOPS 4.11 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.24 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,690.38 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
l#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:1931:1931] [20230204:213603:1675575363:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:37:49 STEP  744 H= 744:00 +CPU= 62.728
  05:38:29 STEP  745 H= 745:00 +CPU= 37.715
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0333
(argv4) batchid: 968
(argv5) wuid: 12184977
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0a2293733ba4099fbe133f2b8f3de008 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
ic_ancil_file: ic_ancil_12184977
ifsdata_file: ifsdata_12184977
climate_data_file: clim_data_12184977
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_300ad98e3e727469f31e1eaa7b8b5640 to: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_422ffaf33a1d0a51301a373adc594014 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12184977.xml
-- Model is restarting --
Adjusting last_iter, 746, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
last_cpu_time: 29984
upload_file_number: 31
last_iter: 743
last_upload: 2678400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363
The child process has been launched with process id: 4901
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:4901:4901] [20230205:054025:1675604425:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:42:18 STEP  744 H= 744:00 +CPU= 50.316
  05:42:48 STEP  745 H= 745:00 +CPU= 27.517
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:44:02 STEP  746 H= 746:00 +CPU= 34.760
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:45:15 STEP  747 H= 747:00 +CPU= 53.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:47:20 STEP  748 H= 748:00 +CPU= 35.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:48:01 STEP  749 H= 749:00 +CPU= 27.430
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:49:02 STEP  750 H= 750:00 +CPU= 45.400
  05:49:25 STEP  751 H= 751:00 +CPU= 22.645
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:50:40 STEP  752 H= 752:00 +CPU= 36.033
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:52:24 STEP  753 H= 753:00 +CPU= 52.850
  05:53:00 STEP  754 H= 754:00 +CPU= 31.912
  05:53:28 STEP  755 H= 755:00 +CPU= 26.608
  05:54:24 STEP  756 H= 756:00 +CPU= 51.716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000756
  05:54:50 STEP  757 H= 757:00 +CPU= 24.870
  05:55:25 STEP  758 H= 758:00 +CPU= 32.383
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:57:32 STEP  759 H= 759:00 +CPU= 48.395
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:58:45 STEP  760 H= 760:00 +CPU= 35.204
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:59:46 STEP  761 H= 761:00 +CPU= 34.361
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:01:43 STEP  762 H= 762:00 +CPU= 46.963
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:37 STEP  763 H= 763:00 +CPU= 34.771
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:04:22 STEP  764 H= 764:00 +CPU= 31.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:05:27 STEP  765 H= 765:00 +CPU= 49.461
  06:05:58 STEP  766 H= 766:00 +CPU= 28.495
  06:06:31 STEP  767 H= 767:00 +CPU= 31.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:07:55 STEP  768 H= 768:00 +CPU= 56.121
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363_31.zip
Uploading the intermediate file: upload_file_31.zip
  06:08:32 STEP  769 H= 769:00 +CPU= 31.311
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:09:23 STEP  770 H= 770:00 +CPU= 28.366
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0333
(argv4) batchid: 968
(argv5) wuid: 12184977
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0a2293733ba4099fbe133f2b8f3de008 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
ic_ancil_file: ic_ancil_12184977
ifsdata_file: ifsdata_12184977
climate_data_file: clim_data_12184977
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_300ad98e3e727469f31e1eaa7b8b5640 to: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_422ffaf33a1d0a51301a373adc594014 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12184977.xml
-- Model is restarting --
Adjusting last_iter, 771, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
last_cpu_time: 31006
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363
The child process has been launched with process id: 8656
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:8656:8656] [20230205:061113:1675606273:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:13:59 STEP  768 H= 768:00 +CPU= 63.159
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:15:24 STEP  769 H= 769:00 +CPU= 35.104
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:16:09 STEP  770 H= 770:00 +CPU= 30.857
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:18:30 STEP  771 H= 771:00 +CPU= 50.590
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:19:30 STEP  772 H= 772:00 +CPU= 33.413
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0333
(argv4) batchid: 968
(argv5) wuid: 12184977
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0a2293733ba4099fbe133f2b8f3de008 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
ic_ancil_file: ic_ancil_12184977
ifsdata_file: ifsdata_12184977
climate_data_file: clim_data_12184977
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_300ad98e3e727469f31e1eaa7b8b5640 to: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_422ffaf33a1d0a51301a373adc594014 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12184977.xml
-- Model is restarting --
Adjusting last_iter, 773, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
last_cpu_time: 31223
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363
The child process has been launched with process id: 10678
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:10678:10678] [20230205:062138:1675606898:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:23:23 STEP  768 H= 768:00 +CPU= 59.613
  06:23:52 STEP  769 H= 769:00 +CPU= 27.281
  06:24:22 STEP  770 H= 770:00 +CPU= 28.609
  06:25:11 STEP  771 H= 771:00 +CPU= 44.506
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:25:55 STEP  772 H= 772:00 +CPU= 30.928
  06:26:23 STEP  773 H= 773:00 +CPU= 26.871
  06:27:10 STEP  774 H= 774:00 +CPU= 44.234
  06:27:41 STEP  775 H= 775:00 +CPU= 29.172
  06:28:09 STEP  776 H= 776:00 +CPU= 26.987
  06:28:57 STEP  777 H= 777:00 +CPU= 43.859
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:29:44 STEP  778 H= 778:00 +CPU= 32.647
  06:30:10 STEP  779 H= 779:00 +CPU= 25.824
  06:30:52 STEP  780 H= 780:00 +CPU= 41.557
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780
  06:31:20 STEP  781 H= 781:00 +CPU= 27.814
  06:31:44 STEP  782 H= 782:00 +CPU= 23.554
  06:32:19 STEP  783 H= 783:00 +CPU= 34.704
  06:32:45 STEP  784 H= 784:00 +CPU= 25.523
  06:33:07 STEP  785 H= 785:00 +CPU= 22.388
  06:33:47 STEP  786 H= 786:00 +CPU= 39.633
  06:34:10 STEP  787 H= 787:00 +CPU= 23.077
  06:34:33 STEP  788 H= 788:00 +CPU= 22.878
  06:35:13 STEP  789 H= 789:00 +CPU= 39.150
  06:35:36 STEP  790 H= 790:00 +CPU= 23.015
  06:35:59 STEP  791 H= 791:00 +CPU= 23.144
  06:36:46 STEP  792 H= 792:00 +CPU= 46.101
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363_32.zip
Uploading the intermediate file: upload_file_32.zip
  06:37:08 STEP  793 H= 793:00 +CPU= 22.524
  06:37:31 STEP  794 H= 794:00 +CPU= 23.206
  06:38:11 STEP  795 H= 795:00 +CPU= 39.303
  06:38:34 STEP  796 H= 796:00 +CPU= 23.090
  06:38:58 STEP  797 H= 797:00 +CPU= 24.128
  06:39:38 STEP  798 H= 798:00 +CPU= 38.855
  06:40:00 STEP  799 H= 799:00 +CPU= 22.437
  06:40:27 STEP  800 H= 800:00 +CPU= 26.398
  06:41:04 STEP  801 H= 801:00 +CPU= 36.313
  06:41:27 STEP  802 H= 802:00 +CPU= 23.044
  06:41:54 STEP  803 H= 803:00 +CPU= 27.133
  06:42:33 STEP  804 H= 804:00 +CPU= 38.479
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000804
  06:42:56 STEP  805 H= 805:00 +CPU= 23.452
  06:43:24 STEP  806 H= 806:00 +CPU= 27.605
  06:43:59 STEP  807 H= 807:00 +CPU= 34.982
  06:44:25 STEP  808 H= 808:00 +CPU= 25.197
  06:44:50 STEP  809 H= 809:00 +CPU= 25.369
  06:45:26 STEP  810 H= 810:00 +CPU= 35.586
  06:45:51 STEP  811 H= 811:00 +CPU= 24.894
  06:46:14 STEP  812 H= 812:00 +CPU= 23.142
  06:46:50 STEP  813 H= 813:00 +CPU= 35.621
  06:47:18 STEP  814 H= 814:00 +CPU= 27.619
  06:47:41 STEP  815 H= 815:00 +CPU= 23.147
  06:48:29 STEP  816 H= 816:00 +CPU= 46.830
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363_33.zip
Uploading the intermediate file: upload_file_33.zip
  06:48:54 STEP  817 H= 817:00 +CPU= 25.483
  06:49:17 STEP  818 H= 818:00 +CPU= 22.916
  06:49:57 STEP  819 H= 819:00 +CPU= 38.971
  06:50:20 STEP  820 H= 820:00 +CPU= 23.535
  06:50:43 STEP  821 H= 821:00 +CPU= 22.693
  06:51:22 STEP  822 H= 822:00 +CPU= 38.941
  06:51:45 STEP  823 H= 823:00 +CPU= 22.817
  06:52:08 STEP  824 H= 824:00 +CPU= 23.148
  06:52:46 STEP  825 H= 825:00 +CPU= 37.152
  06:53:09 STEP  826 H= 826:00 +CPU= 23.272
  06:53:35 STEP  827 H= 827:00 +CPU= 25.920
  06:54:16 STEP  828 H= 828:00 +CPU= 41.355
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000828
  06:54:39 STEP  829 H= 829:00 +CPU= 22.510
  06:55:07 STEP  830 H= 830:00 +CPU= 27.795
  06:55:42 STEP  831 H= 831:00 +CPU= 35.171
  06:56:07 STEP  832 H= 832:00 +CPU= 24.778
  06:56:33 STEP  833 H= 833:00 +CPU= 25.763
  06:57:09 STEP  834 H= 834:00 +CPU= 35.861
  06:57:35 STEP  835 H= 835:00 +CPU= 25.260
  06:58:01 STEP  836 H= 836:00 +CPU= 25.781
  06:58:36 STEP  837 H= 837:00 +CPU= 34.954
  06:59:04 STEP  838 H= 838:00 +CPU= 27.856
  06:59:27 STEP  839 H= 839:00 +CPU= 23.241
  07:00:10 STEP  840 H= 840:00 +CPU= 42.663
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363_34.zip
Uploading the intermediate file: upload_file_34.zip
  07:00:37 STEP  841 H= 841:00 +CPU= 26.964
  07:01:00 STEP  842 H= 842:00 +CPU= 22.929
  07:01:39 STEP  843 H= 843:00 +CPU= 38.119
  07:02:04 STEP  844 H= 844:00 +CPU= 24.801
  07:02:26 STEP  845 H= 845:00 +CPU= 22.222
  07:03:06 STEP  846 H= 846:00 +CPU= 39.954
  07:03:29 STEP  847 H= 847:00 +CPU= 22.612
  07:03:52 STEP  848 H= 848:00 +CPU= 22.863
  07:04:31 STEP  849 H= 849:00 +CPU= 39.186
  07:04:54 STEP  850 H= 850:00 +CPU= 22.898
  07:05:16 STEP  851 H= 851:00 +CPU= 22.183
  07:06:00 STEP  852 H= 852:00 +CPU= 43.529
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000852
  07:06:23 STEP  853 H= 853:00 +CPU= 22.557
  07:06:46 STEP  854 H= 854:00 +CPU= 22.933
  07:07:25 STEP  855 H= 855:00 +CPU= 38.920
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:08:35 STEP  856 H= 856:00 +CPU= 33.878
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:09:43 STEP  857 H= 857:00 +CPU= 32.861
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:10:43 STEP  858 H= 858:00 +CPU= 45.348
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:11:25 STEP  859 H= 859:00 +CPU= 29.485
  07:11:52 STEP  860 H= 860:00 +CPU= 25.735
  07:12:34 STEP  861 H= 861:00 +CPU= 41.169
  07:12:59 STEP  862 H= 862:00 +CPU= 24.893
  07:13:23 STEP  863 H= 863:00 +CPU= 23.162
  07:14:21 STEP  864 H= 864:00 +CPU= 53.876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363_35.zip
Uploading the intermediate file: upload_file_35.zip
  07:14:53 STEP  865 H= 865:00 +CPU= 29.339
  07:15:26 STEP  866 H= 866:00 +CPU= 30.986
  07:16:10 STEP  867 H= 867:00 +CPU= 41.022
  07:16:41 STEP  868 H= 868:00 +CPU= 29.200
  07:17:05 STEP  869 H= 869:00 +CPU= 24.390
  07:17:53 STEP  870 H= 870:00 +CPU= 44.350
  07:18:21 STEP  871 H= 871:00 +CPU= 26.730
  07:18:52 STEP  872 H= 872:00 +CPU= 28.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:20:03 STEP  873 H= 873:00 +CPU= 44.725
  07:20:32 STEP  874 H= 874:00 +CPU= 27.575
  07:21:03 STEP  875 H= 875:00 +CPU= 28.837
  07:21:51 STEP  876 H= 876:00 +CPU= 45.090
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000876
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0333
(argv4) batchid: 968
(argv5) wuid: 12184977
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0a2293733ba4099fbe133f2b8f3de008 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0333_1999050100_123_968_12184977.zip
ic_ancil_file: ic_ancil_12184977
ifsdata_file: ifsdata_12184977
climate_data_file: clim_data_12184977
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_300ad98e3e727469f31e1eaa7b8b5640 to: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12184977.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_422ffaf33a1d0a51301a373adc594014 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12184977.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12184977.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12184977.xml
-- Model is restarting --
Adjusting last_iter, 877, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12184977.xml
last_cpu_time: 34590
upload_file_number: 36
last_iter: 863
last_upload: 3110400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363
The child process has been launched with process id: 12358
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ruly-Inspiron-5558:1:1:12358:12358] [20230205:072315:1675610595:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:24:34 STEP  864 H= 864:00 +CPU= 48.891
  07:25:03 STEP  865 H= 865:00 +CPU= 27.958
  07:25:31 STEP  866 H= 866:00 +CPU= 26.895
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:26:50 STEP  867 H= 867:00 +CPU= 43.953
  07:27:21 STEP  868 H= 868:00 +CPU= 28.613
  07:27:50 STEP  869 H= 869:00 +CPU= 27.723
  07:28:41 STEP  870 H= 870:00 +CPU= 47.506
  07:29:11 STEP  871 H= 871:00 +CPU= 27.651
  07:29:39 STEP  872 H= 872:00 +CPU= 28.085
  07:30:15 STEP  873 H= 873:00 +CPU= 35.604
  07:30:41 STEP  874 H= 874:00 +CPU= 25.239
  07:31:07 STEP  875 H= 875:00 +CPU= 25.445
  07:31:47 STEP  876 H= 876:00 +CPU= 39.726
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000876
  07:32:15 STEP  877 H= 877:00 +CPU= 28.145
  07:32:39 STEP  878 H= 878:00 +CPU= 23.237
  07:33:15 STEP  879 H= 879:00 +CPU= 35.684
  07:33:40 STEP  880 H= 880:00 +CPU= 25.277
  07:34:03 STEP  881 H= 881:00 +CPU= 22.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:35:04 STEP  882 H= 882:00 +CPU= 45.960
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:35:48 STEP  883 H= 883:00 +CPU= 30.916
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:36:38 STEP  884 H= 884:00 +CPU= 33.639
  07:37:30 STEP  885 H= 885:00 +CPU= 47.265
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:38:18 STEP  886 H= 886:00 +CPU= 33.663
  07:38:50 STEP  887 H= 887:00 +CPU= 29.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:41:18 STEP  888 H= 888:00 +CPU= 57.914
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184977/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0333_1999050100_123_968_12184977_0_r2137061363_36.zip
Uploading the intermediate file: upload_file_36.zip
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:47:10 1509518 22270260 oifs_43r3_ps_0333_1999050100_123_968_12184977_0 2,505,600 26,896 0.0107
09 May 2023 16:33:21 1509518 22270260 oifs_43r3_ps_0333_1999050100_123_968_12184977_0 1,382,400 11,839 0.0086


©2024 cpdn.org