<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
2769 H=2769:00 +CPU= 28.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:10:52 STEP 2770 H=2770:00 +CPU= 18.375
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:11:12 STEP 2771 H=2771:00 +CPU= 17.760
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:11:48 STEP 2772 H=2772:00 +CPU= 31.857
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002772
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002772
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:12:07 STEP 2773 H=2773:00 +CPU= 17.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:12:28 STEP 2774 H=2774:00 +CPU= 18.375
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:12:58 STEP 2775 H=2775:00 +CPU= 28.424
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:13:19 STEP 2776 H=2776:00 +CPU= 18.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:13:38 STEP 2777 H=2777:00 +CPU= 17.687
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:14:11 STEP 2778 H=2778:00 +CPU= 28.841
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:14:31 STEP 2779 H=2779:00 +CPU= 17.722
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:14:51 STEP 2780 H=2780:00 +CPU= 18.297
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:15:22 STEP 2781 H=2781:00 +CPU= 28.267
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:15:42 STEP 2782 H=2782:00 +CPU= 18.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:16:02 STEP 2783 H=2783:00 +CPU= 18.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:16:39 STEP 2784 H=2784:00 +CPU= 33.087
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002784
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002784
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002784
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002760
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002760
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002760
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002772
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002772
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002772
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_115.zip
Uploading the intermediate file: upload_file_115.zip
18:16:59 STEP 2785 H=2785:00 +CPU= 17.746
18:17:17 STEP 2786 H=2786:00 +CPU= 18.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:17:50 STEP 2787 H=2787:00 +CPU= 28.528
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:18:11 STEP 2788 H=2788:00 +CPU= 18.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0466
(argv4) batchid: 964
(argv5) wuid: 12181110
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/nvme/scratch1/boinc/slots/2
Project directory is: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110
..mkdir for temp folder for results failed
Copying: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_929f12428e0c91ca2751b38b0023bb85 to: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_0466_1995050100_123_964_12181110.zip
Unzipping the namelist zip file: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_0466_1995050100_123_964_12181110.zip
ic_ancil_file: ic_ancil_12181110
ifsdata_file: ifsdata_12181110
climate_data_file: clim_data_12181110
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b72e40ec8fd2f2389d9315827ae03a7b to: /mnt/nvme/scratch1/boinc/slots/2/ic_ancil_12181110.zip
Unzipping the IC ancils zip file: /mnt/nvme/scratch1/boinc/slots/2/ic_ancil_12181110.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_95586015123d5a8c45211d46553e8133 to: /mnt/nvme/scratch1/boinc/slots/2/ifsdata/ifsdata_12181110.zip
Unzipping the ifsdata_zip file: /mnt/nvme/scratch1/boinc/slots/2/ifsdata/ifsdata_12181110.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/nvme/scratch1/boinc/slots/2/159l_2/clim_data_12181110.zip
Unzipping the climate data zip file: /mnt/nvme/scratch1/boinc/slots/2/159l_2/clim_data_12181110.zip
Checking for progress XML file: /mnt/nvme/scratch1/boinc/slots/2/progress_file_12181110.xml
Opened progress file ok : /mnt/nvme/scratch1/boinc/slots/2/progress_file_12181110.xml
-- Model is restarting --
Adjusting last_iter, 2789, to previous model restart step.
Creating progress file: /mnt/nvme/scratch1/boinc/slots/2/progress_file_12181110.xml
last_cpu_time: 59669
upload_file_number: 116
last_iter: 2783
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100
The child process has been launched with process id: 16045
Executing the command: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:16045:16045] [20221228:190517:1672250717:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:05:54 STEP 2784 H=2784:00 +CPU= 30.873
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:06:14 STEP 2785 H=2785:00 +CPU= 17.921
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:06:34 STEP 2786 H=2786:00 +CPU= 18.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:07:07 STEP 2787 H=2787:00 +CPU= 28.358
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:07:27 STEP 2788 H=2788:00 +CPU= 18.224
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:07:47 STEP 2789 H=2789:00 +CPU= 17.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:08:18 STEP 2790 H=2790:00 +CPU= 28.944
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:08:38 STEP 2791 H=2791:00 +CPU= 17.885
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:08:58 STEP 2792 H=2792:00 +CPU= 18.469
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:09:31 STEP 2793 H=2793:00 +CPU= 28.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:09:51 STEP 2794 H=2794:00 +CPU= 18.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:10:11 STEP 2795 H=2795:00 +CPU= 17.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:10:47 STEP 2796 H=2796:00 +CPU= 31.843
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002796
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002796
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:11:07 STEP 2797 H=2797:00 +CPU= 17.982
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:11:27 STEP 2798 H=2798:00 +CPU= 17.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:11:57 STEP 2799 H=2799:00 +CPU= 28.578
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:12:18 STEP 2800 H=2800:00 +CPU= 18.407
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:12:38 STEP 2801 H=2801:00 +CPU= 17.924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:13:11 STEP 2802 H=2802:00 +CPU= 29.018
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:13:31 STEP 2803 H=2803:00 +CPU= 17.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:13:51 STEP 2804 H=2804:00 +CPU= 18.368
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:14:22 STEP 2805 H=2805:00 +CPU= 28.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:14:42 STEP 2806 H=2806:00 +CPU= 18.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:15:04 STEP 2807 H=2807:00 +CPU= 18.324
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:15:40 STEP 2808 H=2808:00 +CPU= 33.160
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002808
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002808
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002808
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002784
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002784
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002784
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002796
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002796
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002796
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_116.zip
Uploading the intermediate file: upload_file_116.zip
19:15:58 STEP 2809 H=2809:00 +CPU= 17.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:16:18 STEP 2810 H=2810:00 +CPU= 18.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:16:51 STEP 2811 H=2811:00 +CPU= 28.546
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:17:11 STEP 2812 H=2812:00 +CPU= 18.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:17:31 STEP 2813 H=2813:00 +CPU= 17.849
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:18:05 STEP 2814 H=2814:00 +CPU= 29.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:18:25 STEP 2815 H=2815:00 +CPU= 17.992
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:18:45 STEP 2816 H=2816:00 +CPU= 18.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:19:16 STEP 2817 H=2817:00 +CPU= 28.595
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:19:36 STEP 2818 H=2818:00 +CPU= 18.444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:19:56 STEP 2819 H=2819:00 +CPU= 17.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:20:32 STEP 2820 H=2820:00 +CPU= 31.476
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002820
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002820
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002820
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:20:52 STEP 2821 H=2821:00 +CPU= 17.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:21:12 STEP 2822 H=2822:00 +CPU= 18.371
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:21:44 STEP 2823 H=2823:00 +CPU= 28.375
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:22:05 STEP 2824 H=2824:00 +CPU= 18.418
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:22:25 STEP 2825 H=2825:00 +CPU= 17.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:22:55 STEP 2826 H=2826:00 +CPU= 28.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:23:15 STEP 2827 H=2827:00 +CPU= 17.654
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:23:35 STEP 2828 H=2828:00 +CPU= 18.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:24:08 STEP 2829 H=2829:00 +CPU= 28.407
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:24:28 STEP 2830 H=2830:00 +CPU= 18.293
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:24:48 STEP 2831 H=2831:00 +CPU= 18.257
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:25:26 STEP 2832 H=2832:00 +CPU= 33.405
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002832
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002832
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002832
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002808
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002808
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002808
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002820
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002820
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002820
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_117.zip
Uploading the intermediate file: upload_file_117.zip
19:25:44 STEP 2833 H=2833:00 +CPU= 17.793
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:26:05 STEP 2834 H=2834:00 +CPU= 18.320
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:26:35 STEP 2835 H=2835:00 +CPU= 28.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:26:55 STEP 2836 H=2836:00 +CPU= 18.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:27:15 STEP 2837 H=2837:00 +CPU= 17.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:27:48 STEP 2838 H=2838:00 +CPU= 29.026
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:28:08 STEP 2839 H=2839:00 +CPU= 17.920
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:28:29 STEP 2840 H=2840:00 +CPU= 18.447
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:29:00 STEP 2841 H=2841:00 +CPU= 28.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:29:20 STEP 2842 H=2842:00 +CPU= 18.493
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:29:40 STEP 2843 H=2843:00 +CPU= 17.943
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:30:16 STEP 2844 H=2844:00 +CPU= 31.719
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002844
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002844
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:30:36 STEP 2845 H=2845:00 +CPU= 17.864
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:30:56 STEP 2846 H=2846:00 +CPU= 18.452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:31:29 STEP 2847 H=2847:00 +CPU= 28.565
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:31:49 STEP 2848 H=2848:00 +CPU= 18.052
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:32:08 STEP 2849 H=2849:00 +CPU= 17.311
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:32:34 STEP 2850 H=2850:00 +CPU= 23.969
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:32:51 STEP 2851 H=2851:00 +CPU= 15.172
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:33:11 STEP 2852 H=2852:00 +CPU= 17.714
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:33:39 STEP 2853 H=2853:00 +CPU= 26.234
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:34:00 STEP 2854 H=2854:00 +CPU= 18.395
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:34:20 STEP 2855 H=2855:00 +CPU= 18.379
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:34:58 STEP 2856 H=2856:00 +CPU= 33.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002856
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002856
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002856
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002832
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002832
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002832
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002844
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002844
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002844
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_118.zip
Uploading the intermediate file: upload_file_118.zip
19:35:19 STEP 2857 H=2857:00 +CPU= 17.831
19:35:37 STEP 2858 H=2858:00 +CPU= 18.246
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:36:10 STEP 2859 H=2859:00 +CPU= 28.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:36:30 STEP 2860 H=2860:00 +CPU= 18.331
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:36:50 STEP 2861 H=2861:00 +CPU= 17.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:37:23 STEP 2862 H=2862:00 +CPU= 29.149
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:37:43 STEP 2863 H=2863:00 +CPU= 18.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:38:03 STEP 2864 H=2864:00 +CPU= 18.282
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:38:34 STEP 2865 H=2865:00 +CPU= 28.396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:38:54 STEP 2866 H=2866:00 +CPU= 18.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:39:14 STEP 2867 H=2867:00 +CPU= 17.800
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:39:50 STEP 2868 H=2868:00 +CPU= 31.850
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002868
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002868
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002868
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:40:10 STEP 2869 H=2869:00 +CPU= 17.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:40:30 STEP 2870 H=2870:00 +CPU= 18.426
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:41:01 STEP 2871 H=2871:00 +CPU= 28.571
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:41:23 STEP 2872 H=2872:00 +CPU= 18.375
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:41:43 STEP 2873 H=2873:00 +CPU= 17.921
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:42:15 STEP 2874 H=2874:00 +CPU= 29.367
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:42:35 STEP 2875 H=2875:00 +CPU= 17.907
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:42:55 STEP 2876 H=2876:00 +CPU= 18.408
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:43:28 STEP 2877 H=2877:00 +CPU= 28.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:43:48 STEP 2878 H=2878:00 +CPU= 18.327
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:44:08 STEP 2879 H=2879:00 +CPU= 18.396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:44:46 STEP 2880 H=2880:00 +CPU= 33.418
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002880
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002880
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002880
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002856
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002856
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002856
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002868
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002868
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002868
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_119.zip
Uploading the intermediate file: upload_file_119.zip
19:45:04 STEP 2881 H=2881:00 +CPU= 17.772
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:45:25 STEP 2882 H=2882:00 +CPU= 18.365
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:45:55 STEP 2883 H=2883:00 +CPU= 28.376
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:46:16 STEP 2884 H=2884:00 +CPU= 18.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:46:35 STEP 2885 H=2885:00 +CPU= 17.788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:47:08 STEP 2886 H=2886:00 +CPU= 28.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:47:28 STEP 2887 H=2887:00 +CPU= 17.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:47:49 STEP 2888 H=2888:00 +CPU= 18.408
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:48:19 STEP 2889 H=2889:00 +CPU= 28.186
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:48:39 STEP 2890 H=2890:00 +CPU= 18.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:48:59 STEP 2891 H=2891:00 +CPU= 17.997
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:49:35 STEP 2892 H=2892:00 +CPU= 31.777
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002892
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002892
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002892
19:49:55 STEP 2893 H=2893:00 +CPU= 17.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:50:15 STEP 2894 H=2894:00 +CPU= 18.316
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:50:48 STEP 2895 H=2895:00 +CPU= 28.393
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:51:08 STEP 2896 H=2896:00 +CPU= 17.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:51:27 STEP 2897 H=2897:00 +CPU= 17.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:51:58 STEP 2898 H=2898:00 +CPU= 28.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:52:18 STEP 2899 H=2899:00 +CPU= 17.947
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:52:38 STEP 2900 H=2900:00 +CPU= 18.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:53:11 STEP 2901 H=2901:00 +CPU= 28.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:53:31 STEP 2902 H=2902:00 +CPU= 18.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:53:52 STEP 2903 H=2903:00 +CPU= 18.352
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:54:29 STEP 2904 H=2904:00 +CPU= 32.987
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002904
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002904
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002904
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002880
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002880
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002880
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002892
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002892
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002892
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_120.zip
Uploading the intermediate file: upload_file_120.zip
19:54:46 STEP 2905 H=2905:00 +CPU= 17.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:55:07 STEP 2906 H=2906:00 +CPU= 17.954
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:55:37 STEP 2907 H=2907:00 +CPU= 28.138
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:55:57 STEP 2908 H=2908:00 +CPU= 18.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:56:17 STEP 2909 H=2909:00 +CPU= 17.969
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:56:50 STEP 2910 H=2910:00 +CPU= 29.106
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:57:11 STEP 2911 H=2911:00 +CPU= 17.974
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:57:31 STEP 2912 H=2912:00 +CPU= 18.276
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0466
(argv4) batchid: 964
(argv5) wuid: 12181110
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/nvme/scratch1/boinc/slots/2
Project directory is: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110
..mkdir for temp folder for results failed
Copying: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_929f12428e0c91ca2751b38b0023bb85 to: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_0466_1995050100_123_964_12181110.zip
Unzipping the namelist zip file: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_ps_0466_1995050100_123_964_12181110.zip
ic_ancil_file: ic_ancil_12181110
ifsdata_file: ifsdata_12181110
climate_data_file: clim_data_12181110
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b72e40ec8fd2f2389d9315827ae03a7b to: /mnt/nvme/scratch1/boinc/slots/2/ic_ancil_12181110.zip
Unzipping the IC ancils zip file: /mnt/nvme/scratch1/boinc/slots/2/ic_ancil_12181110.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_95586015123d5a8c45211d46553e8133 to: /mnt/nvme/scratch1/boinc/slots/2/ifsdata/ifsdata_12181110.zip
Unzipping the ifsdata_zip file: /mnt/nvme/scratch1/boinc/slots/2/ifsdata/ifsdata_12181110.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/nvme/scratch1/boinc/slots/2/159l_2/clim_data_12181110.zip
Unzipping the climate data zip file: /mnt/nvme/scratch1/boinc/slots/2/159l_2/clim_data_12181110.zip
Checking for progress XML file: /mnt/nvme/scratch1/boinc/slots/2/progress_file_12181110.xml
Opened progress file ok : /mnt/nvme/scratch1/boinc/slots/2/progress_file_12181110.xml
-- Model is restarting --
Adjusting last_iter, 2913, to previous model restart step.
Creating progress file: /mnt/nvme/scratch1/boinc/slots/2/progress_file_12181110.xml
last_cpu_time: 62490
upload_file_number: 121
last_iter: 2903
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100
The child process has been launched with process id: 25334
Executing the command: /mnt/nvme/scratch1/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:25334:25334] [20221228:201612:1672254972:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:16:48 STEP 2904 H=2904:00 +CPU= 30.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:17:07 STEP 2905 H=2905:00 +CPU= 17.018
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:17:26 STEP 2906 H=2906:00 +CPU= 17.516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:17:56 STEP 2907 H=2907:00 +CPU= 27.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:18:16 STEP 2908 H=2908:00 +CPU= 17.507
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:18:34 STEP 2909 H=2909:00 +CPU= 16.733
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:19:07 STEP 2910 H=2910:00 +CPU= 28.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:19:26 STEP 2911 H=2911:00 +CPU= 17.058
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:19:45 STEP 2912 H=2912:00 +CPU= 17.745
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:20:15 STEP 2913 H=2913:00 +CPU= 27.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:20:35 STEP 2914 H=2914:00 +CPU= 17.551
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:20:54 STEP 2915 H=2915:00 +CPU= 17.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:21:29 STEP 2916 H=2916:00 +CPU= 30.730
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002916
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002916
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002916
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:21:48 STEP 2917 H=2917:00 +CPU= 17.134
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:22:07 STEP 2918 H=2918:00 +CPU= 17.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:22:37 STEP 2919 H=2919:00 +CPU= 27.619
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:22:57 STEP 2920 H=2920:00 +CPU= 17.583
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:23:16 STEP 2921 H=2921:00 +CPU= 17.067
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:23:48 STEP 2922 H=2922:00 +CPU= 28.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:24:07 STEP 2923 H=2923:00 +CPU= 17.065
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:24:27 STEP 2924 H=2924:00 +CPU= 17.537
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:24:56 STEP 2925 H=2925:00 +CPU= 27.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:25:16 STEP 2926 H=2926:00 +CPU= 17.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:25:35 STEP 2927 H=2927:00 +CPU= 17.538
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:26:11 STEP 2928 H=2928:00 +CPU= 31.790
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002928
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002928
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002928
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002904
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002904
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002904
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002916
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002916
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002916
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_121.zip
Uploading the intermediate file: upload_file_121.zip
20:26:28 STEP 2929 H=2929:00 +CPU= 16.983
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:26:48 STEP 2930 H=2930:00 +CPU= 17.647
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:27:18 STEP 2931 H=2931:00 +CPU= 27.734
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:27:37 STEP 2932 H=2932:00 +CPU= 17.489
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:27:56 STEP 2933 H=2933:00 +CPU= 17.058
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:28:29 STEP 2934 H=2934:00 +CPU= 28.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:28:48 STEP 2935 H=2935:00 +CPU= 17.119
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:29:08 STEP 2936 H=2936:00 +CPU= 17.547
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:29:37 STEP 2937 H=2937:00 +CPU= 27.586
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:29:57 STEP 2938 H=2938:00 +CPU= 17.601
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:30:16 STEP 2939 H=2939:00 +CPU= 17.085
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:30:51 STEP 2940 H=2940:00 +CPU= 30.714
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002940
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002940
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002940
20:31:10 STEP 2941 H=2941:00 +CPU= 17.054
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:31:29 STEP 2942 H=2942:00 +CPU= 17.594
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:31:55 STEP 2943 H=2943:00 +CPU= 23.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:32:14 STEP 2944 H=2944:00 +CPU= 16.429
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:32:33 STEP 2945 H=2945:00 +CPU= 17.116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:33:05 STEP 2946 H=2946:00 +CPU= 28.177
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:33:24 STEP 2947 H=2947:00 +CPU= 17.043
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:33:44 STEP 2948 H=2948:00 +CPU= 17.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:34:13 STEP 2949 H=2949:00 +CPU= 27.632
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:34:33 STEP 2950 H=2950:00 +CPU= 17.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:34:52 STEP 2951 H=2951:00 +CPU= 17.667
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:35:28 STEP 2952 H=2952:00 +CPU= 31.137
The child process terminated with status: 0
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMGGhq0f+002952
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMSHhq0f+002952
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/2/ICMUAhq0f+002952
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002928
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002928
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002952
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002952
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002928
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002940
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMSHhq0f+002940
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMUAhq0f+002940
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181110/ICMGGhq0f+002952
Zipping up the final file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0466_1995050100_123_964_12181110_0_r779770100_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
20:38:50 (25298): called boinc_finish(0)
</stderr_txt>
]]>
