<core_client_version>7.22.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a39r
(argv4) batchid: 1018
(argv5) wuid: 12292566
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/robbie/slots/10
Project directory is: /home/robbie/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/robbie/projects/climateprediction.net/oifs_43r3_bl_12292566
Copying: /home/robbie/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/robbie/slots/10/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/robbie/slots/10/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_edf91f87766ca052bed2ca23c78dac92 to: /home/robbie/slots/10/oifs_43r3_bl_a39r_2016092300_20_1018_12292566.zip
Unzipping the namelist zip file: /home/robbie/slots/10/oifs_43r3_bl_a39r_2016092300_20_1018_12292566.zip
ic_ancil_file: ic_ancil_12292566
ifsdata_file: ifsdata_12292566
climate_data_file: clim_data_12292566
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/robbie/slots/10/ic_ancil_12292566.zip
Unzipping the IC ancils zip file: /home/robbie/slots/10/ic_ancil_12292566.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3a905dabc4a444461a7f8f449b270696 to: /home/robbie/slots/10/ifsdata/ifsdata_12292566.zip
Unzipping the ifsdata_zip file: /home/robbie/slots/10/ifsdata/ifsdata_12292566.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/robbie/slots/10/159l_2/clim_data_12292566.zip
Unzipping the climate data zip file: /home/robbie/slots/10/159l_2/clim_data_12292566.zip
Checking for progress XML file: /home/robbie/slots/10/progress_file_12292566.xml
Creating progress file: /home/robbie/slots/10/progress_file_12292566.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a39r_2016092300_20_1018_12292566_0_r1767549011
The child process has been launched with process id: 1581
Executing the command: /home/robbie/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at 0x1)
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:robbie-Intel-Z490:1:1:1581:1581] [20240613:201405:1718306045:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
20:17:26 STEP 0 H= 0:00 +CPU= 18.184
Moving to projects directory: /home/robbie/slots/10/ICMGGh7zg+000000
Moving to projects directory: /home/robbie/slots/10/ICMSHh7zg+000000
Moving to projects directory: /home/robbie/slots/10/ICMUAh7zg+000000
21:50:50 STEP 1 H= 0:15 +CPU= 24.266
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:55:03 STEP 2 H= 0:30 +CPU= 21.598
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:55:38 STEP 3 H= 0:45 +CPU= 12.377
21:55:54 STEP 4 H= 1:00 +CPU= 16.412
21:56:15 STEP 5 H= 1:15 +CPU= 14.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:13:38 STEP 6 H= 1:30 +CPU= 19.448
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
NSTEP = 0 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 379743156 BYTES IN 6 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0264( 0.8%)
NSTEP = 0 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+000000
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0167 SECONDS 42359.2 Mbytes/s, TOTAL TIME= 0.1532(10.9%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 5248584000 BYTES IN 125 RECORDS TRANSFERRED IN 0.0052 SECONDS******** Mbytes/s, TOTAL TIME= 0.0812( 6.4%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0002 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0023( 8.7%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 319456771 BYTES IN 4 RECORDS TRANSFERRED IN 0.0005 SECONDS638913.5 Mbytes/s, TOTAL TIME= 0.0038(13.2%)
STEPO_FPOS WAS : T00000000M
NSTEP = 0 STEPO 0AAA00AAA
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 328036500 BYTES IN 7 RECORDS TRANSFERRED IN 0.1771 SECONDS 1852.3 Mbytes/s, TOTAL TIME= 5.8795( 3.0%)
20:17:26 STEP 0 H= 0:00 +CPU= 18.184
NSTEP = 1 STEPO 0AAA00AAA
21:50:50 STEP 1 H= 0:15 +CPU= 24.266
NSTEP = 2 STEPO 0AAA00AAA
21:55:03 STEP 2 H= 0:30 +CPU= 21.598
NSTEP = 3 STEPO 0AAA00AAA
21:55:38 STEP 3 H= 0:45 +CPU= 12.377
NSTEP = 4 STEPO 0AAA00AAA
21:55:54 STEP 4 H= 1:00 +CPU= 16.412
ISTEP= 5 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP= 5 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
NSTEP = 5 STEPO 0AAA00AAA
21:56:15 STEP 5 H= 1:15 +CPU= 14.040
NSTEP = 6 STEPO 0AAA00AAA
22:13:38 STEP 6 H= 1:30 +CPU= 19.448
NSTEP = 7 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
20:17:28 0AAA00AAA STEPO 1 14.220 14.220 192.503 0:21 3:22 0.55515332309566E-07 2GB 0MB
21:50:50 0AAA00AAA STEPO 2 24.268 24.268 5602.189 0:46 96:44 0.55515332309566E-07 2GB 0MB
21:55:03 0AAA00AAA STEPO 3 21.599 21.599 253.065 1:07 100:57 0.55515332309566E-07 2GB 0MB
21:55:38 0AAA00AAA STEPO 4 12.377 12.377 34.915 1:20 101:32 0.55515332309566E-07 2GB 0MB
21:55:54 0AAA00AAA STEPO 5 16.413 16.413 16.445 1:36 101:49 0.55515332309566E-07 2GB 0MB
21:56:15 0AAA00AAA STEPO 6 14.040 14.040 20.754 1:50 102:10 0.55515332309566E-07 2GB 0MB
22:13:38 0AAA00AAA STEPO 7 19.449 19.449 1042.671 2:10 119:32 0.55515332309566E-07 2GB 0MB
------------------------------------------------
>>> Printing last 8 lines from file: /home/robbie/slots/10/progress_file_12292566.xml
<running_values>
<last_cpu_time>138.960000</last_cpu_time>
<upload_file_number>0</upload_file_number>
<last_iter>7</last_iter>
<last_upload>0</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
