<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939_93.zip
Uploading the intermediate file: upload_file_93.zip
22:30:51 STEP 2257 H=2257:00 +CPU= 18.804
22:31:11 STEP 2258 H=2258:00 +CPU= 18.429
22:31:41 STEP 2259 H=2259:00 +CPU= 28.022
22:32:00 STEP 2260 H=2260:00 +CPU= 18.448
22:32:20 STEP 2261 H=2261:00 +CPU= 18.207
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:33:03 STEP 2262 H=2262:00 +CPU= 30.231
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:33:35 STEP 2263 H=2263:00 +CPU= 19.214
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:34:06 STEP 2264 H=2264:00 +CPU= 18.832
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:34:48 STEP 2265 H=2265:00 +CPU= 29.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:35:20 STEP 2266 H=2266:00 +CPU= 19.550
22:35:40 STEP 2267 H=2267:00 +CPU= 18.912
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:36:38 STEP 2268 H=2268:00 +CPU= 33.189
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:37:21 STEP 2269 H=2269:00 +CPU= 19.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:38:04 STEP 2270 H=2270:00 +CPU= 19.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:39:38 STEP 2271 H=2271:00 +CPU= 29.395
22:39:56 STEP 2272 H=2272:00 +CPU= 17.514
22:40:14 STEP 2273 H=2273:00 +CPU= 17.067
22:40:42 STEP 2274 H=2274:00 +CPU= 27.510
22:41:00 STEP 2275 H=2275:00 +CPU= 16.998
22:41:18 STEP 2276 H=2276:00 +CPU= 17.485
22:41:46 STEP 2277 H=2277:00 +CPU= 26.814
22:42:04 STEP 2278 H=2278:00 +CPU= 17.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:42:45 STEP 2279 H=2279:00 +CPU= 19.312
22:43:19 STEP 2280 H=2280:00 +CPU= 32.206
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939_94.zip
Uploading the intermediate file: upload_file_94.zip
22:43:38 STEP 2281 H=2281:00 +CPU= 17.478
22:43:57 STEP 2282 H=2282:00 +CPU= 18.217
22:44:27 STEP 2283 H=2283:00 +CPU= 28.279
22:44:45 STEP 2284 H=2284:00 +CPU= 18.079
22:45:04 STEP 2285 H=2285:00 +CPU= 17.533
22:45:34 STEP 2286 H=2286:00 +CPU= 28.552
22:45:52 STEP 2287 H=2287:00 +CPU= 17.282
22:46:11 STEP 2288 H=2288:00 +CPU= 17.996
22:46:41 STEP 2289 H=2289:00 +CPU= 28.687
22:47:00 STEP 2290 H=2290:00 +CPU= 18.167
22:47:19 STEP 2291 H=2291:00 +CPU= 18.139
22:47:52 STEP 2292 H=2292:00 +CPU= 31.104
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002292
22:48:10 STEP 2293 H=2293:00 +CPU= 17.219
22:48:30 STEP 2294 H=2294:00 +CPU= 19.115
22:49:00 STEP 2295 H=2295:00 +CPU= 28.401
22:49:19 STEP 2296 H=2296:00 +CPU= 18.362
22:49:38 STEP 2297 H=2297:00 +CPU= 17.873
22:50:08 STEP 2298 H=2298:00 +CPU= 28.695
22:50:28 STEP 2299 H=2299:00 +CPU= 18.173
22:50:46 STEP 2300 H=2300:00 +CPU= 17.762
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:52:39 STEP 2301 H=2301:00 +CPU= 28.151
22:52:59 STEP 2302 H=2302:00 +CPU= 17.806
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:53:52 STEP 2303 H=2303:00 +CPU= 20.838
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:54:42 STEP 2304 H=2304:00 +CPU= 36.034
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939_95.zip
Uploading the intermediate file: upload_file_95.zip
22:55:02 STEP 2305 H=2305:00 +CPU= 18.370
22:55:21 STEP 2306 H=2306:00 +CPU= 18.114
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.160] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.171] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.171] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.171] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.171] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.171] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.171] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS CLDPP T/S (1002)
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : TURBULENCE_LAYER
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFOUTER
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFMAIN
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225543:1673474143:2689.173] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFEXCU
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225544:1673474144:2690.173] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2883:2883] [20230111:225544:1673474144:2690.173] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000FCBB8E vdfexcu_ 428 vdfexcu.F90
oifs_43r3_model.e 0000000000FAB617 vdfmain_ 981 vdfmain.F90
oifs_43r3_model.e 0000000000F92699 vdfouter_ 739 vdfouter.F90
oifs_43r3_model.e 0000000000F88512 turbulence_layer_ 158 turbulence_layer.F90
oifs_43r3_model.e 0000000000F0B683 callpar_ 869 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0999
(argv4) batchid: 951
(argv5) wuid: 12168643
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_50368b8c5df880bb2615e92463b5422e to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0999_1982050100_123_951_12168643.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0999_1982050100_123_951_12168643.zip
ic_ancil_file: ic_ancil_12168643
ifsdata_file: ifsdata_12168643
climate_data_file: clim_data_12168643
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d45d9291126066868fb815ed51225022 to: /var/lib/boinc-client/slots/0/ic_ancil_12168643.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168643.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f90d7952681da85af247141f188e1b9b to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168643.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168643.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168643.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168643.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168643.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168643.xml
-- Model is restarting --
Adjusting last_iter, 2307, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168643.xml
last_cpu_time: 50965
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939
The child process has been launched with process id: 2141
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:220841:1673557721:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:11:30 STEP 2304 H=2304:00 +CPU= 36.856
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:12:01 STEP 2305 H=2305:00 +CPU= 18.741
22:12:21 STEP 2306 H=2306:00 +CPU= 18.944
22:12:52 STEP 2307 H=2307:00 +CPU= 28.977
22:13:12 STEP 2308 H=2308:00 +CPU= 18.858
22:13:32 STEP 2309 H=2309:00 +CPU= 18.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:14:16 STEP 2310 H=2310:00 +CPU= 30.317
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:14:48 STEP 2311 H=2311:00 +CPU= 19.087
22:15:09 STEP 2312 H=2312:00 +CPU= 19.464
22:15:41 STEP 2313 H=2313:00 +CPU= 29.933
22:16:02 STEP 2314 H=2314:00 +CPU= 19.098
22:16:22 STEP 2315 H=2315:00 +CPU= 18.619
22:16:56 STEP 2316 H=2316:00 +CPU= 32.167
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002316
22:17:16 STEP 2317 H=2317:00 +CPU= 18.592
22:17:37 STEP 2318 H=2318:00 +CPU= 19.612
22:18:09 STEP 2319 H=2319:00 +CPU= 29.575
22:18:29 STEP 2320 H=2320:00 +CPU= 19.133
22:18:50 STEP 2321 H=2321:00 +CPU= 19.216
22:19:24 STEP 2322 H=2322:00 +CPU= 31.359
22:19:44 STEP 2323 H=2323:00 +CPU= 19.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:20:17 STEP 2324 H=2324:00 +CPU= 20.244
22:20:49 STEP 2325 H=2325:00 +CPU= 30.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:21:21 STEP 2326 H=2326:00 +CPU= 20.033
22:21:43 STEP 2327 H=2327:00 +CPU= 19.800
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:22:31 STEP 2328 H=2328:00 +CPU= 34.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939_96.zip
Uploading the intermediate file: upload_file_96.zip
22:23:05 STEP 2329 H=2329:00 +CPU= 20.011
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:23:38 STEP 2330 H=2330:00 +CPU= 20.532
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:24:32 STEP 2331 H=2331:00 +CPU= 30.130
22:24:53 STEP 2332 H=2332:00 +CPU= 19.729
22:25:13 STEP 2333 H=2333:00 +CPU= 19.046
22:25:46 STEP 2334 H=2334:00 +CPU= 30.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:28:33 STEP 2335 H=2335:00 +CPU= 21.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:29:57 STEP 2336 H=2336:00 +CPU= 20.217
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:31:35 STEP 2337 H=2337:00 +CPU= 30.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:33:09 STEP 2338 H=2338:00 +CPU= 20.782
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.766] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001)
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.767] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.767] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.767] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLIMAER_LAYER
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223456:1673559296:1574.767] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADACA
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223457:1673559297:1575.767] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2141:2141] [20230112:223457:1673559297:1575.767] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001CCB1D1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000FFE8C9 radaca_ 224 radaca.F90
oifs_43r3_model.e 0000000000FF616B climaer_layer_ 120 climaer_layer.F90
oifs_43r3_model.e 0000000000F09C63 callpar_ 787 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0999
(argv4) batchid: 951
(argv5) wuid: 12168643
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_50368b8c5df880bb2615e92463b5422e to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0999_1982050100_123_951_12168643.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0999_1982050100_123_951_12168643.zip
ic_ancil_file: ic_ancil_12168643
ifsdata_file: ifsdata_12168643
climate_data_file: clim_data_12168643
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d45d9291126066868fb815ed51225022 to: /var/lib/boinc-client/slots/0/ic_ancil_12168643.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168643.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f90d7952681da85af247141f188e1b9b to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168643.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168643.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168643.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168643.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168643.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168643.xml
-- Model is restarting --
Adjusting last_iter, 2339, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168643.xml
last_cpu_time: 51800
upload_file_number: 97
last_iter: 2327
last_upload: 8380800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939
The child process has been launched with process id: 2092
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062027:1673587227:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
06:21:48 STEP 2328 H=2328:00 +CPU= 32.772
double free or corruption (top)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3993MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3993 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001)
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUD_LAYER
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062204:1673587324:97.146] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUDSC
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062205:1673587325:98.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Olymp:1:1:2092:2092] [20230113:062205:1673587325:98.146] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001CCAC6B Unknown Unknown Unknown
oifs_43r3_model.e 00000000010228DC cloudsc_ 1145 cloudsc.F90
oifs_43r3_model.e 0000000001014D57 cloud_layer_ 131 cloud_layer.F90
oifs_43r3_model.e 0000000000F1A790 callpar_ 1223 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0999
(argv4) batchid: 951
(argv5) wuid: 12168643
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_50368b8c5df880bb2615e92463b5422e to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0999_1982050100_123_951_12168643.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0999_1982050100_123_951_12168643.zip
ic_ancil_file: ic_ancil_12168643
ifsdata_file: ifsdata_12168643
climate_data_file: clim_data_12168643
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d45d9291126066868fb815ed51225022 to: /var/lib/boinc-client/slots/0/ic_ancil_12168643.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168643.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f90d7952681da85af247141f188e1b9b to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168643.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168643.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168643.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168643.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168643.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168643.xml
-- Model is restarting --
Adjusting last_iter, 2329, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168643.xml
last_cpu_time: 51843
upload_file_number: 97
last_iter: 2327
last_upload: 8380800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939
The child process has been launched with process id: 2080
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Olymp:1:1:2080:2080] [20230114:152325:1673706205:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:24:35 STEP 2328 H=2328:00 +CPU= 32.301
15:24:54 STEP 2329 H=2329:00 +CPU= 17.898
15:25:14 STEP 2330 H=2330:00 +CPU= 18.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:26:46 STEP 2331 H=2331:00 +CPU= 29.003
15:27:04 STEP 2332 H=2332:00 +CPU= 17.496
15:27:21 STEP 2333 H=2333:00 +CPU= 16.979
15:27:50 STEP 2334 H=2334:00 +CPU= 27.599
15:28:08 STEP 2335 H=2335:00 +CPU= 17.625
15:28:26 STEP 2336 H=2336:00 +CPU= 17.845
15:28:55 STEP 2337 H=2337:00 +CPU= 27.499
15:29:13 STEP 2338 H=2338:00 +CPU= 17.799
15:29:31 STEP 2339 H=2339:00 +CPU= 17.317
15:30:03 STEP 2340 H=2340:00 +CPU= 30.681
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002340
15:30:22 STEP 2341 H=2341:00 +CPU= 17.697
15:30:42 STEP 2342 H=2342:00 +CPU= 18.561
15:31:11 STEP 2343 H=2343:00 +CPU= 27.915
15:31:29 STEP 2344 H=2344:00 +CPU= 17.946
15:31:47 STEP 2345 H=2345:00 +CPU= 17.359
15:32:17 STEP 2346 H=2346:00 +CPU= 28.206
15:32:35 STEP 2347 H=2347:00 +CPU= 17.448
15:32:53 STEP 2348 H=2348:00 +CPU= 17.916
15:33:22 STEP 2349 H=2349:00 +CPU= 27.647
15:33:41 STEP 2350 H=2350:00 +CPU= 17.882
15:33:59 STEP 2351 H=2351:00 +CPU= 17.930
15:34:33 STEP 2352 H=2352:00 +CPU= 32.086
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168643/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0999_1982050100_123_951_12168643_0_r24487939_97.zip
Uploading the intermediate file: upload_file_97.zip
15:34:53 STEP 2353 H=2353:00 +CPU= 19.159
15:35:13 STEP 2354 H=2354:00 +CPU= 19.114
15:35:43 STEP 2355 H=2355:00 +CPU= 28.204
15:36:06 STEP 2356 H=2356:00 +CPU= 21.212
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:37:06 STEP 2357 H=2357:00 +CPU= 18.150
15:37:37 STEP 2358 H=2358:00 +CPU= 28.880
15:37:56 STEP 2359 H=2359:00 +CPU= 17.877
15:38:14 STEP 2360 H=2360:00 +CPU= 17.713
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
