<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
04:20 STEP 1073 H=1073:00 +CPU= 68.775
20:06:32 STEP 1074 H=1074:00 +CPU=114.572
20:07:51 STEP 1075 H=1075:00 +CPU= 68.530
20:09:11 STEP 1076 H=1076:00 +CPU= 71.057
20:11:19 STEP 1077 H=1077:00 +CPU=111.434
20:12:44 STEP 1078 H=1078:00 +CPU= 71.472
20:14:06 STEP 1079 H=1079:00 +CPU= 71.618
20:16:39 STEP 1080 H=1080:00 +CPU=130.738
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_44.zip
Uploading the intermediate file: upload_file_44.zip
20:18:01 STEP 1081 H=1081:00 +CPU= 69.096
20:19:24 STEP 1082 H=1082:00 +CPU= 71.795
20:21:35 STEP 1083 H=1083:00 +CPU=112.016
20:22:59 STEP 1084 H=1084:00 +CPU= 71.795
20:24:17 STEP 1085 H=1085:00 +CPU= 68.933
20:26:30 STEP 1086 H=1086:00 +CPU=115.148
20:27:48 STEP 1087 H=1087:00 +CPU= 69.152
20:29:09 STEP 1088 H=1088:00 +CPU= 72.081
20:31:20 STEP 1089 H=1089:00 +CPU=112.224
20:32:42 STEP 1090 H=1090:00 +CPU= 71.753
20:34:04 STEP 1091 H=1091:00 +CPU= 69.045
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0321
(argv4) batchid: 983
(argv5) wuid: 12199965
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
ic_ancil_file: ic_ancil_12199965
ifsdata_file: ifsdata_12199965
climate_data_file: clim_data_12199965
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml
-- Model is restarting --
Adjusting last_iter, 1092, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
last_cpu_time: 101990
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732
The child process has been launched with process id: 1277610
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:44:14 STEP 1080 H=1080:00 +CPU=126.610
21:45:34 STEP 1081 H=1081:00 +CPU= 68.614
21:46:56 STEP 1082 H=1082:00 +CPU= 71.349
21:48:59 STEP 1083 H=1083:00 +CPU=110.733
21:50:22 STEP 1084 H=1084:00 +CPU= 71.495
21:51:40 STEP 1085 H=1085:00 +CPU= 68.509
21:53:49 STEP 1086 H=1086:00 +CPU=114.404
21:55:10 STEP 1087 H=1087:00 +CPU= 69.124
21:56:32 STEP 1088 H=1088:00 +CPU= 71.774
21:58:45 STEP 1089 H=1089:00 +CPU=112.297
22:00:11 STEP 1090 H=1090:00 +CPU= 72.019
22:01:34 STEP 1091 H=1091:00 +CPU= 69.361
22:04:01 STEP 1092 H=1092:00 +CPU=125.346
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001092
22:05:22 STEP 1093 H=1093:00 +CPU= 69.186
22:06:42 STEP 1094 H=1094:00 +CPU= 71.741
22:08:51 STEP 1095 H=1095:00 +CPU=112.172
22:10:14 STEP 1096 H=1096:00 +CPU= 71.934
22:11:33 STEP 1097 H=1097:00 +CPU= 69.016
22:13:45 STEP 1098 H=1098:00 +CPU=114.569
22:15:04 STEP 1099 H=1099:00 +CPU= 68.785
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:16:40 STEP 1100 H=1100:00 +CPU= 71.593
22:18:48 STEP 1101 H=1101:00 +CPU=111.326
22:20:08 STEP 1102 H=1102:00 +CPU= 71.447
22:21:29 STEP 1103 H=1103:00 +CPU= 71.146
22:23:57 STEP 1104 H=1104:00 +CPU=129.718
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_45.zip
Uploading the intermediate file: upload_file_45.zip
22:25:17 STEP 1105 H=1105:00 +CPU= 68.479
22:26:40 STEP 1106 H=1106:00 +CPU= 71.438
22:28:49 STEP 1107 H=1107:00 +CPU=111.583
22:30:12 STEP 1108 H=1108:00 +CPU= 71.387
22:31:35 STEP 1109 H=1109:00 +CPU= 69.092
22:33:51 STEP 1110 H=1110:00 +CPU=114.920
22:35:10 STEP 1111 H=1111:00 +CPU= 68.852
22:36:35 STEP 1112 H=1112:00 +CPU= 71.677
22:38:43 STEP 1113 H=1113:00 +CPU=111.919
22:40:06 STEP 1114 H=1114:00 +CPU= 72.022
22:41:27 STEP 1115 H=1115:00 +CPU= 69.181
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0321
(argv4) batchid: 983
(argv5) wuid: 12199965
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
ic_ancil_file: ic_ancil_12199965
ifsdata_file: ifsdata_12199965
climate_data_file: clim_data_12199965
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml
-- Model is restarting --
Adjusting last_iter, 1116, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
last_cpu_time: 105091
upload_file_number: 46
last_iter: 1103
last_upload: 3974400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732
The child process has been launched with process id: 1282947
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
23:34:20 STEP 1104 H=1104:00 +CPU=127.004
23:35:39 STEP 1105 H=1105:00 +CPU= 68.733
23:37:01 STEP 1106 H=1106:00 +CPU= 71.374
23:39:12 STEP 1107 H=1107:00 +CPU=111.489
23:40:35 STEP 1108 H=1108:00 +CPU= 71.736
23:41:56 STEP 1109 H=1109:00 +CPU= 68.959
23:44:10 STEP 1110 H=1110:00 +CPU=114.942
23:45:32 STEP 1111 H=1111:00 +CPU= 69.162
23:46:55 STEP 1112 H=1112:00 +CPU= 71.795
23:49:08 STEP 1113 H=1113:00 +CPU=112.110
23:50:31 STEP 1114 H=1114:00 +CPU= 71.938
23:51:50 STEP 1115 H=1115:00 +CPU= 68.948
23:54:10 STEP 1116 H=1116:00 +CPU=125.098
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001116
23:55:32 STEP 1117 H=1117:00 +CPU= 68.934
23:56:55 STEP 1118 H=1118:00 +CPU= 71.430
23:59:03 STEP 1119 H=1119:00 +CPU=111.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:01:30 STEP 1120 H=1120:00 +CPU= 71.545
00:02:48 STEP 1121 H=1121:00 +CPU= 68.449
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:07:04 STEP 1122 H=1122:00 +CPU=114.705
00:08:29 STEP 1123 H=1123:00 +CPU= 68.973
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:10:55 STEP 1124 H=1124:00 +CPU= 71.602
00:13:05 STEP 1125 H=1125:00 +CPU=111.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:15:40 STEP 1126 H=1126:00 +CPU= 72.016
00:17:05 STEP 1127 H=1127:00 +CPU= 71.438
00:19:44 STEP 1128 H=1128:00 +CPU=130.999
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_46.zip
Uploading the intermediate file: upload_file_46.zip
00:21:06 STEP 1129 H=1129:00 +CPU= 68.934
00:22:29 STEP 1130 H=1130:00 +CPU= 71.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
00:26:41 STEP 1131 H=1131:00 +CPU=112.189
00:28:03 STEP 1132 H=1132:00 +CPU= 71.528
00:29:22 STEP 1133 H=1133:00 +CPU= 68.799
00:31:32 STEP 1134 H=1134:00 +CPU=114.664
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0321
(argv4) batchid: 983
(argv5) wuid: 12199965
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
ic_ancil_file: ic_ancil_12199965
ifsdata_file: ifsdata_12199965
climate_data_file: clim_data_12199965
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml
-- Model is restarting --
Adjusting last_iter, 1135, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
last_cpu_time: 107824
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732
The child process has been launched with process id: 1289273
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
01:35:55 STEP 1128 H=1128:00 +CPU=128.528
01:37:15 STEP 1129 H=1129:00 +CPU= 69.386
01:38:38 STEP 1130 H=1130:00 +CPU= 72.139
01:40:52 STEP 1131 H=1131:00 +CPU=112.774
01:42:16 STEP 1132 H=1132:00 +CPU= 72.069
01:43:38 STEP 1133 H=1133:00 +CPU= 69.339
01:45:54 STEP 1134 H=1134:00 +CPU=116.032
01:47:15 STEP 1135 H=1135:00 +CPU= 69.879
01:48:41 STEP 1136 H=1136:00 +CPU= 72.683
01:50:46 STEP 1137 H=1137:00 +CPU=112.591
01:52:11 STEP 1138 H=1138:00 +CPU= 72.677
01:53:32 STEP 1139 H=1139:00 +CPU= 69.986
01:56:00 STEP 1140 H=1140:00 +CPU=126.149
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001140
01:57:24 STEP 1141 H=1141:00 +CPU= 69.990
01:58:50 STEP 1142 H=1142:00 +CPU= 72.742
02:01:04 STEP 1143 H=1143:00 +CPU=113.497
02:02:28 STEP 1144 H=1144:00 +CPU= 72.642
02:03:47 STEP 1145 H=1145:00 +CPU= 69.701
02:06:02 STEP 1146 H=1146:00 +CPU=116.521
02:07:22 STEP 1147 H=1147:00 +CPU= 69.715
02:08:47 STEP 1148 H=1148:00 +CPU= 72.547
02:10:58 STEP 1149 H=1149:00 +CPU=112.880
02:12:22 STEP 1150 H=1150:00 +CPU= 72.668
02:13:46 STEP 1151 H=1151:00 +CPU= 72.434
02:16:17 STEP 1152 H=1152:00 +CPU=132.394
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_47.zip
Uploading the intermediate file: upload_file_47.zip
02:17:41 STEP 1153 H=1153:00 +CPU= 70.119
02:19:03 STEP 1154 H=1154:00 +CPU= 72.736
02:21:12 STEP 1155 H=1155:00 +CPU=113.221
02:22:36 STEP 1156 H=1156:00 +CPU= 72.686
02:23:52 STEP 1157 H=1157:00 +CPU= 69.699
02:26:06 STEP 1158 H=1158:00 +CPU=116.132
02:27:28 STEP 1159 H=1159:00 +CPU= 70.041
02:28:52 STEP 1160 H=1160:00 +CPU= 72.865
02:31:05 STEP 1161 H=1161:00 +CPU=113.531
02:32:32 STEP 1162 H=1162:00 +CPU= 72.911
02:33:51 STEP 1163 H=1163:00 +CPU= 70.279
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0321
(argv4) batchid: 983
(argv5) wuid: 12199965
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
ic_ancil_file: ic_ancil_12199965
ifsdata_file: ifsdata_12199965
climate_data_file: clim_data_12199965
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml
-- Model is restarting --
Adjusting last_iter, 1164, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
last_cpu_time: 110965
upload_file_number: 48
last_iter: 1151
last_upload: 4147200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732
The child process has been launched with process id: 1295296
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
03:39:45 STEP 1152 H=1152:00 +CPU=128.512
03:41:07 STEP 1153 H=1153:00 +CPU= 69.646
03:42:31 STEP 1154 H=1154:00 +CPU= 72.417
03:44:44 STEP 1155 H=1155:00 +CPU=113.041
03:46:11 STEP 1156 H=1156:00 +CPU= 72.578
03:47:33 STEP 1157 H=1157:00 +CPU= 69.595
03:49:48 STEP 1158 H=1158:00 +CPU=116.160
03:51:08 STEP 1159 H=1159:00 +CPU= 69.575
03:52:31 STEP 1160 H=1160:00 +CPU= 72.422
03:54:42 STEP 1161 H=1161:00 +CPU=112.878
03:56:08 STEP 1162 H=1162:00 +CPU= 72.129
03:57:30 STEP 1163 H=1163:00 +CPU= 69.430
03:59:51 STEP 1164 H=1164:00 +CPU=125.303
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001164
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:01:24 STEP 1165 H=1165:00 +CPU= 69.702
04:02:49 STEP 1166 H=1166:00 +CPU= 72.557
04:05:02 STEP 1167 H=1167:00 +CPU=112.958
04:06:23 STEP 1168 H=1168:00 +CPU= 72.186
04:07:44 STEP 1169 H=1169:00 +CPU= 69.322
04:09:59 STEP 1170 H=1170:00 +CPU=115.940
04:11:22 STEP 1171 H=1171:00 +CPU= 69.534
04:12:46 STEP 1172 H=1172:00 +CPU= 72.334
04:14:58 STEP 1173 H=1173:00 +CPU=112.600
04:16:24 STEP 1174 H=1174:00 +CPU= 72.335
04:17:50 STEP 1175 H=1175:00 +CPU= 72.342
04:20:21 STEP 1176 H=1176:00 +CPU=131.611
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_48.zip
Uploading the intermediate file: upload_file_48.zip
04:21:43 STEP 1177 H=1177:00 +CPU= 69.447
04:23:06 STEP 1178 H=1178:00 +CPU= 72.523
04:25:15 STEP 1179 H=1179:00 +CPU=112.703
04:26:39 STEP 1180 H=1180:00 +CPU= 72.197
04:28:01 STEP 1181 H=1181:00 +CPU= 69.579
04:30:14 STEP 1182 H=1182:00 +CPU=115.585
04:31:35 STEP 1183 H=1183:00 +CPU= 69.297
04:33:00 STEP 1184 H=1184:00 +CPU= 72.177
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:35:44 STEP 1185 H=1185:00 +CPU=112.101
04:37:13 STEP 1186 H=1186:00 +CPU= 72.748
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0321
(argv4) batchid: 983
(argv5) wuid: 12199965
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip
ic_ancil_file: ic_ancil_12199965
ifsdata_file: ifsdata_12199965
climate_data_file: clim_data_12199965
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml
-- Model is restarting --
Adjusting last_iter, 1187, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml
last_cpu_time: 114034
upload_file_number: 49
last_iter: 1175
last_upload: 4233600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732
The child process has been launched with process id: 1306053
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:56:34 STEP 1176 H=1176:00 +CPU=126.717
06:57:49 STEP 1177 H=1177:00 +CPU= 68.580
06:59:11 STEP 1178 H=1178:00 +CPU= 71.400
07:01:21 STEP 1179 H=1179:00 +CPU=111.671
07:02:46 STEP 1180 H=1180:00 +CPU= 71.550
07:04:03 STEP 1181 H=1181:00 +CPU= 68.887
07:06:15 STEP 1182 H=1182:00 +CPU=114.518
07:07:35 STEP 1183 H=1183:00 +CPU= 68.778
07:08:58 STEP 1184 H=1184:00 +CPU= 71.686
07:11:07 STEP 1185 H=1185:00 +CPU=111.922
07:12:30 STEP 1186 H=1186:00 +CPU= 71.882
07:13:53 STEP 1187 H=1187:00 +CPU= 68.913
07:16:19 STEP 1188 H=1188:00 +CPU=124.903
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001188
07:17:38 STEP 1189 H=1189:00 +CPU= 69.038
07:19:00 STEP 1190 H=1190:00 +CPU= 71.778
07:21:07 STEP 1191 H=1191:00 +CPU=111.629
07:22:27 STEP 1192 H=1192:00 +CPU= 71.586
07:23:47 STEP 1193 H=1193:00 +CPU= 68.798
07:25:56 STEP 1194 H=1194:00 +CPU=114.075
07:27:15 STEP 1195 H=1195:00 +CPU= 68.547
07:28:36 STEP 1196 H=1196:00 +CPU= 71.327
07:30:44 STEP 1197 H=1197:00 +CPU=111.396
07:32:03 STEP 1198 H=1198:00 +CPU= 71.367
07:33:26 STEP 1199 H=1199:00 +CPU= 71.275
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
