<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221_91.zip
Uploading the intermediate file: upload_file_91.zip
02:32:19 STEP 2209 H=2209:00 +CPU= 19.618
02:32:40 STEP 2210 H=2210:00 +CPU= 19.841
02:33:11 STEP 2211 H=2211:00 +CPU= 30.295
02:33:31 STEP 2212 H=2212:00 +CPU= 19.502
02:33:51 STEP 2213 H=2213:00 +CPU= 19.221
02:34:23 STEP 2214 H=2214:00 +CPU= 31.284
02:34:43 STEP 2215 H=2215:00 +CPU= 19.201
02:35:03 STEP 2216 H=2216:00 +CPU= 19.975
02:35:35 STEP 2217 H=2217:00 +CPU= 30.462
02:35:55 STEP 2218 H=2218:00 +CPU= 19.825
02:36:16 STEP 2219 H=2219:00 +CPU= 19.628
02:36:50 STEP 2220 H=2220:00 +CPU= 33.527
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002220
02:37:10 STEP 2221 H=2221:00 +CPU= 19.406
02:37:31 STEP 2222 H=2222:00 +CPU= 19.781
02:38:03 STEP 2223 H=2223:00 +CPU= 30.980
02:38:24 STEP 2224 H=2224:00 +CPU= 20.174
02:38:44 STEP 2225 H=2225:00 +CPU= 19.257
02:39:16 STEP 2226 H=2226:00 +CPU= 31.114
02:39:36 STEP 2227 H=2227:00 +CPU= 19.137
02:39:57 STEP 2228 H=2228:00 +CPU= 20.024
02:40:28 STEP 2229 H=2229:00 +CPU= 30.680
02:40:48 STEP 2230 H=2230:00 +CPU= 19.529
02:41:10 STEP 2231 H=2231:00 +CPU= 20.874
02:41:48 STEP 2232 H=2232:00 +CPU= 36.569
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221_92.zip
Uploading the intermediate file: upload_file_92.zip
02:42:07 STEP 2233 H=2233:00 +CPU= 19.084
02:42:28 STEP 2234 H=2234:00 +CPU= 19.954
02:42:59 STEP 2235 H=2235:00 +CPU= 30.387
02:43:18 STEP 2236 H=2236:00 +CPU= 18.940
02:43:38 STEP 2237 H=2237:00 +CPU= 18.975
02:44:09 STEP 2238 H=2238:00 +CPU= 30.427
02:44:29 STEP 2239 H=2239:00 +CPU= 19.644
02:44:50 STEP 2240 H=2240:00 +CPU= 19.897
02:45:20 STEP 2241 H=2241:00 +CPU= 29.662
02:45:41 STEP 2242 H=2242:00 +CPU= 20.160
02:46:01 STEP 2243 H=2243:00 +CPU= 19.378
02:46:34 STEP 2244 H=2244:00 +CPU= 32.570
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002244
02:46:54 STEP 2245 H=2245:00 +CPU= 19.436
02:47:14 STEP 2246 H=2246:00 +CPU= 20.065
02:47:45 STEP 2247 H=2247:00 +CPU= 29.828
02:48:05 STEP 2248 H=2248:00 +CPU= 19.433
02:48:25 STEP 2249 H=2249:00 +CPU= 19.376
02:48:57 STEP 2250 H=2250:00 +CPU= 30.947
02:49:17 STEP 2251 H=2251:00 +CPU= 19.548
02:49:37 STEP 2252 H=2252:00 +CPU= 19.875
02:50:09 STEP 2253 H=2253:00 +CPU= 30.412
02:50:29 STEP 2254 H=2254:00 +CPU= 20.028
02:50:50 STEP 2255 H=2255:00 +CPU= 19.847
02:51:29 STEP 2256 H=2256:00 +CPU= 37.927
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221_93.zip
Uploading the intermediate file: upload_file_93.zip
02:51:50 STEP 2257 H=2257:00 +CPU= 19.828
02:52:11 STEP 2258 H=2258:00 +CPU= 20.130
02:52:42 STEP 2259 H=2259:00 +CPU= 30.612
02:53:03 STEP 2260 H=2260:00 +CPU= 20.132
02:53:23 STEP 2261 H=2261:00 +CPU= 19.552
02:53:55 STEP 2262 H=2262:00 +CPU= 30.930
02:54:15 STEP 2263 H=2263:00 +CPU= 19.320
02:54:35 STEP 2264 H=2264:00 +CPU= 19.340
02:55:06 STEP 2265 H=2265:00 +CPU= 30.381
02:55:27 STEP 2266 H=2266:00 +CPU= 20.148
02:55:47 STEP 2267 H=2267:00 +CPU= 19.246
02:56:23 STEP 2268 H=2268:00 +CPU= 34.427
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002268
02:56:43 STEP 2269 H=2269:00 +CPU= 19.588
02:57:03 STEP 2270 H=2270:00 +CPU= 19.971
02:57:35 STEP 2271 H=2271:00 +CPU= 30.542
02:57:56 STEP 2272 H=2272:00 +CPU= 20.379
02:58:16 STEP 2273 H=2273:00 +CPU= 19.347
02:58:49 STEP 2274 H=2274:00 +CPU= 31.322
02:59:09 STEP 2275 H=2275:00 +CPU= 19.706
02:59:30 STEP 2276 H=2276:00 +CPU= 19.989
03:00:03 STEP 2277 H=2277:00 +CPU= 31.079
03:00:24 STEP 2278 H=2278:00 +CPU= 20.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
03:00:59 STEP 2279 H=2279:00 +CPU= 19.971
03:01:38 STEP 2280 H=2280:00 +CPU= 36.565
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221_94.zip
Uploading the intermediate file: upload_file_94.zip
03:01:58 STEP 2281 H=2281:00 +CPU= 19.941
Uploading trickle at timestep: 8208000
03:02:18 STEP 2282 H=2282:00 +CPU= 19.963
03:02:50 STEP 2283 H=2283:00 +CPU= 30.902
03:03:10 STEP 2284 H=2284:00 +CPU= 19.239
03:03:30 STEP 2285 H=2285:00 +CPU= 19.686
03:04:01 STEP 2286 H=2286:00 +CPU= 30.447
03:04:21 STEP 2287 H=2287:00 +CPU= 19.574
03:04:41 STEP 2288 H=2288:00 +CPU= 20.076
03:05:12 STEP 2289 H=2289:00 +CPU= 29.939
03:05:32 STEP 2290 H=2290:00 +CPU= 19.437
03:05:51 STEP 2291 H=2291:00 +CPU= 18.965
03:06:25 STEP 2292 H=2292:00 +CPU= 33.381
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002292
03:06:45 STEP 2293 H=2293:00 +CPU= 19.427
03:07:05 STEP 2294 H=2294:00 +CPU= 19.808
03:07:36 STEP 2295 H=2295:00 +CPU= 29.475
03:07:56 STEP 2296 H=2296:00 +CPU= 19.947
03:08:16 STEP 2297 H=2297:00 +CPU= 19.642
03:08:48 STEP 2298 H=2298:00 +CPU= 30.785
03:09:08 STEP 2299 H=2299:00 +CPU= 19.130
03:09:28 STEP 2300 H=2300:00 +CPU= 19.403
03:10:00 STEP 2301 H=2301:00 +CPU= 29.615
03:10:20 STEP 2302 H=2302:00 +CPU= 19.699
03:10:41 STEP 2303 H=2303:00 +CPU= 19.782
03:11:17 STEP 2304 H=2304:00 +CPU= 35.070
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221_95.zip
Uploading the intermediate file: upload_file_95.zip
03:11:37 STEP 2305 H=2305:00 +CPU= 19.145
03:11:57 STEP 2306 H=2306:00 +CPU= 19.877
03:12:28 STEP 2307 H=2307:00 +CPU= 30.071
03:12:48 STEP 2308 H=2308:00 +CPU= 19.612
03:13:08 STEP 2309 H=2309:00 +CPU= 19.254
03:13:40 STEP 2310 H=2310:00 +CPU= 30.741
03:14:00 STEP 2311 H=2311:00 +CPU= 19.312
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.872] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.877] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.877] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001)
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : TURBULENCE_LAYER
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFOUTER
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFMAIN
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031409:1673576049:16439.880] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CLOUDVAR
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031410:1673576050:16440.881] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2925:2925] [20230113:031410:1673576050:16440.881] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000FD052E cloudvar_ 243 cloudvar.F90
oifs_43r3_model.e 0000000000FB3DC7 vdfmain_ 890 vdfmain.F90
oifs_43r3_model.e 0000000000F92699 vdfouter_ 739 vdfouter.F90
oifs_43r3_model.e 0000000000F88512 turbulence_layer_ 158 turbulence_layer.F90
oifs_43r3_model.e 0000000000F0B683 callpar_ 869 callpar.F90
oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0455
(argv4) batchid: 955
(argv5) wuid: 12172099
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bfbaaca9718998191829719d81a46afb to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0455_1986050100_123_955_12172099.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0455_1986050100_123_955_12172099.zip
ic_ancil_file: ic_ancil_12172099
ifsdata_file: ifsdata_12172099
climate_data_file: clim_data_12172099
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ee7e50ffd42d0c92c73341775dde7464 to: /var/lib/boinc-client/slots/7/ic_ancil_12172099.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12172099.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_76dd4c1808ed5f30ce20d95a89b29008 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12172099.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12172099.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12172099.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12172099.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12172099.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12172099.xml
-- Model is restarting --
Adjusting last_iter, 2312, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12172099.xml
last_cpu_time: 62349
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221
The child process has been launched with process id: 2954
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212159:1674591719:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:22:44 STEP 2304 H=2304:00 +CPU= 29.913
21:22:59 STEP 2305 H=2305:00 +CPU= 14.844
21:23:15 STEP 2306 H=2306:00 +CPU= 15.566
21:23:40 STEP 2307 H=2307:00 +CPU= 24.459
21:23:56 STEP 2308 H=2308:00 +CPU= 15.861
21:24:12 STEP 2309 H=2309:00 +CPU= 15.501
21:24:38 STEP 2310 H=2310:00 +CPU= 25.086
21:24:54 STEP 2311 H=2311:00 +CPU= 15.955
21:25:08 STEP 2312 H=2312:00 +CPU= 14.302
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4032MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
double free or corruption (top)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4032 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADINTG
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADINTG-RADLSW (1210)
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADIATION_SCHEME
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_interface:radiation
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_mcica_lw:solver_mcica_lw
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212515:1674591915:196.080] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_cloud_generator:cloud_generator
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212516:1674591916:197.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2954:2954] [20230124:212516:1674591916:197.080] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 000000000185E437 radiation_pdf_sam 141 radiation_pdf_sampler.F90
oifs_43r3_model.e 000000000196FB80 radiation_cloud_g 198 radiation_cloud_generator.F90
oifs_43r3_model.e 0000000001972EE7 radiation_mcica_l 179 radiation_mcica_lw.F90
oifs_43r3_model.e 0000000001862B6D radiation_interfa 387 radiation_interface.F90
oifs_43r3_model.e 00000000012D30E5 radiation_scheme_ 484 radiation_scheme.F90
oifs_43r3_model.e 000000000129A415 radintg_ 1455 radintg.F90
oifs_43r3_model.e 0000000001270124 raddrv_ 537 raddrv.F90
oifs_43r3_model.e 0000000000BFD839 gp_model_ 482 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0455
(argv4) batchid: 955
(argv5) wuid: 12172099
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bfbaaca9718998191829719d81a46afb to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0455_1986050100_123_955_12172099.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0455_1986050100_123_955_12172099.zip
ic_ancil_file: ic_ancil_12172099
ifsdata_file: ifsdata_12172099
climate_data_file: clim_data_12172099
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ee7e50ffd42d0c92c73341775dde7464 to: /var/lib/boinc-client/slots/7/ic_ancil_12172099.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12172099.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_76dd4c1808ed5f30ce20d95a89b29008 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12172099.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12172099.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12172099.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12172099.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12172099.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12172099.xml
-- Model is restarting --
Adjusting last_iter, 2313, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12172099.xml
last_cpu_time: 62523
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221
The child process has been launched with process id: 2931
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212819:1674592099:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:28:56 STEP 2304 H=2304:00 +CPU= 28.317
21:29:13 STEP 2305 H=2305:00 +CPU= 16.836
21:29:30 STEP 2306 H=2306:00 +CPU= 16.823
double free or corruption (top)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.605] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2549MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2549 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAMODEL
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-WAMODEL 2 (1431)
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : IMPLSCH
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212954:1674592194:94.606] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SNONLIN
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212955:1674592195:95.606] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2931:2931] [20230124:212955:1674592195:95.606] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001A67007 snonlin_ 191 snonlin.F
oifs_43r3_model.e 0000000001A31273 implsch_ 285 implsch.F
oifs_43r3_model.e 000000000168CBD7 wamodel_ 1008 wamodel.F
oifs_43r3_model.e 0000000000872295 wavemdl_ 584 wavemdl.F
oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90
oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0455
(argv4) batchid: 955
(argv5) wuid: 12172099
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bfbaaca9718998191829719d81a46afb to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0455_1986050100_123_955_12172099.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0455_1986050100_123_955_12172099.zip
ic_ancil_file: ic_ancil_12172099
ifsdata_file: ifsdata_12172099
climate_data_file: clim_data_12172099
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ee7e50ffd42d0c92c73341775dde7464 to: /var/lib/boinc-client/slots/7/ic_ancil_12172099.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12172099.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_76dd4c1808ed5f30ce20d95a89b29008 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12172099.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12172099.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12172099.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12172099.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12172099.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12172099.xml
-- Model is restarting --
Adjusting last_iter, 2307, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12172099.xml
last_cpu_time: 62605
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221
The child process has been launched with process id: 2894
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2894:2894] [20230124:213303:1674592383:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:33:48 STEP 2304 H=2304:00 +CPU= 29.283
21:34:04 STEP 2305 H=2305:00 +CPU= 15.391
21:34:19 STEP 2306 H=2306:00 +CPU= 15.055
21:34:44 STEP 2307 H=2307:00 +CPU= 23.812
21:35:00 STEP 2308 H=2308:00 +CPU= 16.418
21:35:17 STEP 2309 H=2309:00 +CPU= 16.540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:37:58 STEP 2310 H=2310:00 +CPU= 32.382
21:38:18 STEP 2311 H=2311:00 +CPU= 19.154
21:38:37 STEP 2312 H=2312:00 +CPU= 18.333
21:39:11 STEP 2313 H=2313:00 +CPU= 32.264
21:39:33 STEP 2314 H=2314:00 +CPU= 19.754
21:39:52 STEP 2315 H=2315:00 +CPU= 18.435
21:40:24 STEP 2316 H=2316:00 +CPU= 31.660
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002316
21:40:42 STEP 2317 H=2317:00 +CPU= 17.632
21:41:01 STEP 2318 H=2318:00 +CPU= 18.849
21:41:30 STEP 2319 H=2319:00 +CPU= 29.027
21:41:50 STEP 2320 H=2320:00 +CPU= 18.769
21:42:09 STEP 2321 H=2321:00 +CPU= 18.780
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:42:44 STEP 2322 H=2322:00 +CPU= 28.843
21:43:02 STEP 2323 H=2323:00 +CPU= 17.442
21:43:18 STEP 2324 H=2324:00 +CPU= 16.339
21:43:44 STEP 2325 H=2325:00 +CPU= 24.847
21:43:59 STEP 2326 H=2326:00 +CPU= 15.661
21:44:14 STEP 2327 H=2327:00 +CPU= 14.609
21:44:41 STEP 2328 H=2328:00 +CPU= 26.666
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172099/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0455_1986050100_123_955_12172099_1_r1689536221_96.zip
Uploading the intermediate file: upload_file_96.zip
21:44:56 STEP 2329 H=2329:00 +CPU= 14.685
21:45:12 STEP 2330 H=2330:00 +CPU= 15.452
21:45:35 STEP 2331 H=2331:00 +CPU= 22.807
21:45:50 STEP 2332 H=2332:00 +CPU= 15.405
21:46:06 STEP 2333 H=2333:00 +CPU= 15.347
21:46:30 STEP 2334 H=2334:00 +CPU= 23.843
21:46:46 STEP 2335 H=2335:00 +CPU= 15.575
21:47:03 STEP 2336 H=2336:00 +CPU= 16.406
21:47:27 STEP 2337 H=2337:00 +CPU= 23.799
21:47:43 STEP 2338 H=2338:00 +CPU= 15.628
21:47:59 STEP 2339 H=2339:00 +CPU= 15.362
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
