climateprediction.net home page
Task 22305357

Task 22305357

Name oifs_43r3_ps_0961_1983050100_123_952_12169605_1
Workunit 12169605
Created 4 Feb 2023, 2:08:22 UTC
Sent 4 Feb 2023, 2:08:32 UTC
Report deadline 5 Apr 2023, 2:08:32 UTC
Received 9 Feb 2023, 17:29:22 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1534946
Run time 14 hours 41 min 31 sec
CPU time 14 hours 15 min 11 sec
Validate state Invalid
Credit 11,128.32
Device peak FLOPS 5.41 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.69 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,682.05 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
last_iter: 2279
last_upload: 8208000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245
The child process has been launched with process id: 1841
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:200224:1675731744:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:03:11 STEP 2280 H=2280:00 +CPU= 32.467
  20:03:32 STEP 2281 H=2281:00 +CPU= 18.024
  20:03:57 STEP 2282 H=2282:00 +CPU= 20.729
  20:04:32 STEP 2283 H=2283:00 +CPU= 29.588
  20:04:52 STEP 2284 H=2284:00 +CPU= 18.476
  20:05:13 STEP 2285 H=2285:00 +CPU= 18.184
  20:05:45 STEP 2286 H=2286:00 +CPU= 28.929
  20:06:05 STEP 2287 H=2287:00 +CPU= 17.677
  20:06:26 STEP 2288 H=2288:00 +CPU= 17.882
  20:06:56 STEP 2289 H=2289:00 +CPU= 27.152
  20:07:17 STEP 2290 H=2290:00 +CPU= 18.078
  20:07:36 STEP 2291 H=2291:00 +CPU= 17.956
  20:08:10 STEP 2292 H=2292:00 +CPU= 29.802
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002292
  20:08:29 STEP 2293 H=2293:00 +CPU= 17.497
  20:08:49 STEP 2294 H=2294:00 +CPU= 17.712
  20:09:18 STEP 2295 H=2295:00 +CPU= 26.437
  20:09:38 STEP 2296 H=2296:00 +CPU= 17.686
  20:09:56 STEP 2297 H=2297:00 +CPU= 17.202
  20:10:25 STEP 2298 H=2298:00 +CPU= 26.837
  20:10:43 STEP 2299 H=2299:00 +CPU= 16.696
  20:11:01 STEP 2300 H=2300:00 +CPU= 17.206
  20:11:28 STEP 2301 H=2301:00 +CPU= 25.761
  20:11:44 STEP 2302 H=2302:00 +CPU= 15.858
  20:12:00 STEP 2303 H=2303:00 +CPU= 15.911
  20:12:29 STEP 2304 H=2304:00 +CPU= 28.662
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_95.zip
Uploading the intermediate file: upload_file_95.zip
  20:12:44 STEP 2305 H=2305:00 +CPU= 15.678
  20:13:02 STEP 2306 H=2306:00 +CPU= 16.784
  20:13:30 STEP 2307 H=2307:00 +CPU= 26.121
  20:13:48 STEP 2308 H=2308:00 +CPU= 17.095
  20:14:07 STEP 2309 H=2309:00 +CPU= 16.919
  20:14:35 STEP 2310 H=2310:00 +CPU= 26.406
  20:14:53 STEP 2311 H=2311:00 +CPU= 16.804
  20:15:11 STEP 2312 H=2312:00 +CPU= 17.289
  20:15:39 STEP 2313 H=2313:00 +CPU= 26.266
  20:16:00 STEP 2314 H=2314:00 +CPU= 18.890
  20:16:22 STEP 2315 H=2315:00 +CPU= 20.330
  20:16:58 STEP 2316 H=2316:00 +CPU= 32.459
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002316
  20:17:16 STEP 2317 H=2317:00 +CPU= 16.677
  20:17:35 STEP 2318 H=2318:00 +CPU= 17.307
  20:18:03 STEP 2319 H=2319:00 +CPU= 26.100
  20:18:21 STEP 2320 H=2320:00 +CPU= 17.404
  20:18:40 STEP 2321 H=2321:00 +CPU= 16.853
  20:19:08 STEP 2322 H=2322:00 +CPU= 26.380
  20:19:26 STEP 2323 H=2323:00 +CPU= 17.424
  20:19:45 STEP 2324 H=2324:00 +CPU= 17.666
  20:20:13 STEP 2325 H=2325:00 +CPU= 26.025
  20:20:32 STEP 2326 H=2326:00 +CPU= 17.157
  20:20:50 STEP 2327 H=2327:00 +CPU= 17.276
  20:21:23 STEP 2328 H=2328:00 +CPU= 31.024
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_96.zip
Uploading the intermediate file: upload_file_96.zip
  20:21:41 STEP 2329 H=2329:00 +CPU= 16.802
  20:22:00 STEP 2330 H=2330:00 +CPU= 17.335
  20:22:27 STEP 2331 H=2331:00 +CPU= 25.931
  20:22:45 STEP 2332 H=2332:00 +CPU= 17.328
  20:23:03 STEP 2333 H=2333:00 +CPU= 16.730
  20:23:32 STEP 2334 H=2334:00 +CPU= 26.601
  20:23:50 STEP 2335 H=2335:00 +CPU= 17.088
  20:24:08 STEP 2336 H=2336:00 +CPU= 17.295
  20:24:37 STEP 2337 H=2337:00 +CPU= 26.649
  20:24:55 STEP 2338 H=2338:00 +CPU= 17.284
  20:25:13 STEP 2339 H=2339:00 +CPU= 16.663
  20:25:45 STEP 2340 H=2340:00 +CPU= 29.096
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002340
  20:26:02 STEP 2341 H=2341:00 +CPU= 16.728
  20:26:21 STEP 2342 H=2342:00 +CPU= 17.245
  20:26:49 STEP 2343 H=2343:00 +CPU= 25.915
  20:27:07 STEP 2344 H=2344:00 +CPU= 17.207
  20:27:25 STEP 2345 H=2345:00 +CPU= 17.006
  20:27:54 STEP 2346 H=2346:00 +CPU= 26.536
  20:28:11 STEP 2347 H=2347:00 +CPU= 16.721
  20:28:30 STEP 2348 H=2348:00 +CPU= 17.392
  20:28:58 STEP 2349 H=2349:00 +CPU= 25.914
  20:29:17 STEP 2350 H=2350:00 +CPU= 17.451
  20:29:36 STEP 2351 H=2351:00 +CPU= 17.733
  20:30:09 STEP 2352 H=2352:00 +CPU= 31.169
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_97.zip
Uploading the intermediate file: upload_file_97.zip
  20:30:27 STEP 2353 H=2353:00 +CPU= 16.806
  20:30:46 STEP 2354 H=2354:00 +CPU= 17.477
  20:31:13 STEP 2355 H=2355:00 +CPU= 25.983
  20:31:32 STEP 2356 H=2356:00 +CPU= 17.014
  20:31:50 STEP 2357 H=2357:00 +CPU= 16.934
  20:32:18 STEP 2358 H=2358:00 +CPU= 26.428
  20:32:36 STEP 2359 H=2359:00 +CPU= 16.692
  20:32:54 STEP 2360 H=2360:00 +CPU= 17.508
  20:33:22 STEP 2361 H=2361:00 +CPU= 26.269
  20:33:41 STEP 2362 H=2362:00 +CPU= 17.171
  20:33:59 STEP 2363 H=2363:00 +CPU= 16.871
  20:34:29 STEP 2364 H=2364:00 +CPU= 28.491
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002364
  20:34:49 STEP 2365 H=2365:00 +CPU= 17.921
  20:35:07 STEP 2366 H=2366:00 +CPU= 17.147
  20:35:36 STEP 2367 H=2367:00 +CPU= 26.471
  20:35:55 STEP 2368 H=2368:00 +CPU= 17.831
  20:36:13 STEP 2369 H=2369:00 +CPU= 17.139
  20:36:48 STEP 2370 H=2370:00 +CPU= 29.188
  20:37:08 STEP 2371 H=2371:00 +CPU= 17.551
  20:37:30 STEP 2372 H=2372:00 +CPU= 18.390
  20:38:03 STEP 2373 H=2373:00 +CPU= 27.546
  20:38:24 STEP 2374 H=2374:00 +CPU= 18.252
  20:38:47 STEP 2375 H=2375:00 +CPU= 18.676
  20:39:26 STEP 2376 H=2376:00 +CPU= 33.259
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_98.zip
Uploading the intermediate file: upload_file_98.zip
  20:39:48 STEP 2377 H=2377:00 +CPU= 18.170
  20:40:10 STEP 2378 H=2378:00 +CPU= 18.822
  20:40:44 STEP 2379 H=2379:00 +CPU= 28.139
  20:41:05 STEP 2380 H=2380:00 +CPU= 18.417
  20:41:27 STEP 2381 H=2381:00 +CPU= 18.146
  20:42:01 STEP 2382 H=2382:00 +CPU= 28.471
  20:42:22 STEP 2383 H=2383:00 +CPU= 17.798
  20:42:43 STEP 2384 H=2384:00 +CPU= 18.280
  20:43:17 STEP 2385 H=2385:00 +CPU= 28.172
  20:43:38 STEP 2386 H=2386:00 +CPU= 18.202
  20:43:59 STEP 2387 H=2387:00 +CPU= 18.014
  20:44:35 STEP 2388 H=2388:00 +CPU= 30.616
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002388
  20:44:57 STEP 2389 H=2389:00 +CPU= 18.303
  20:45:16 STEP 2390 H=2390:00 +CPU= 18.496
  20:45:49 STEP 2391 H=2391:00 +CPU= 30.544
  20:46:08 STEP 2392 H=2392:00 +CPU= 17.611
  20:46:27 STEP 2393 H=2393:00 +CPU= 17.363
  20:46:56 STEP 2394 H=2394:00 +CPU= 26.778
  20:47:14 STEP 2395 H=2395:00 +CPU= 16.875
  20:47:33 STEP 2396 H=2396:00 +CPU= 17.502
  20:48:00 STEP 2397 H=2397:00 +CPU= 25.876
  20:48:19 STEP 2398 H=2398:00 +CPU= 17.515
  20:48:37 STEP 2399 H=2399:00 +CPU= 17.259
  20:49:10 STEP 2400 H=2400:00 +CPU= 31.049
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_99.zip
Uploading the intermediate file: upload_file_99.zip
  20:49:28 STEP 2401 H=2401:00 +CPU= 16.851
  20:49:47 STEP 2402 H=2402:00 +CPU= 17.440
  20:50:15 STEP 2403 H=2403:00 +CPU= 26.526
  20:50:34 STEP 2404 H=2404:00 +CPU= 17.291
  20:50:52 STEP 2405 H=2405:00 +CPU= 16.744
  20:51:20 STEP 2406 H=2406:00 +CPU= 26.602
  20:51:38 STEP 2407 H=2407:00 +CPU= 16.834
  20:51:57 STEP 2408 H=2408:00 +CPU= 17.398
  20:52:24 STEP 2409 H=2409:00 +CPU= 26.024
  20:52:43 STEP 2410 H=2410:00 +CPU= 17.833
  20:53:01 STEP 2411 H=2411:00 +CPU= 16.861
  20:53:32 STEP 2412 H=2412:00 +CPU= 29.023
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002412
  20:53:51 STEP 2413 H=2413:00 +CPU= 17.567
  20:54:09 STEP 2414 H=2414:00 +CPU= 17.306
  20:54:37 STEP 2415 H=2415:00 +CPU= 26.159
  20:54:55 STEP 2416 H=2416:00 +CPU= 17.285
  20:55:14 STEP 2417 H=2417:00 +CPU= 17.280
  20:55:42 STEP 2418 H=2418:00 +CPU= 26.744
  20:56:01 STEP 2419 H=2419:00 +CPU= 17.122
  20:56:19 STEP 2420 H=2420:00 +CPU= 17.270
  20:56:46 STEP 2421 H=2421:00 +CPU= 25.907
  20:57:05 STEP 2422 H=2422:00 +CPU= 17.458
  20:57:24 STEP 2423 H=2423:00 +CPU= 17.268
  20:57:57 STEP 2424 H=2424:00 +CPU= 31.237
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_100.zip
Uploading the intermediate file: upload_file_100.zip
  20:58:15 STEP 2425 H=2425:00 +CPU= 17.001
  20:58:33 STEP 2426 H=2426:00 +CPU= 17.377
  20:59:02 STEP 2427 H=2427:00 +CPU= 26.539
  20:59:20 STEP 2428 H=2428:00 +CPU= 17.358
  20:59:38 STEP 2429 H=2429:00 +CPU= 16.853
  21:00:07 STEP 2430 H=2430:00 +CPU= 26.852
  21:00:25 STEP 2431 H=2431:00 +CPU= 16.818
  21:00:43 STEP 2432 H=2432:00 +CPU= 17.232
  21:01:11 STEP 2433 H=2433:00 +CPU= 26.299
  21:01:30 STEP 2434 H=2434:00 +CPU= 17.310
  21:01:49 STEP 2435 H=2435:00 +CPU= 17.478
  21:02:20 STEP 2436 H=2436:00 +CPU= 29.399
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002436
  21:02:36 STEP 2437 H=2437:00 +CPU= 15.786
  21:02:58 STEP 2438 H=2438:00 +CPU= 19.112
  21:03:28 STEP 2439 H=2439:00 +CPU= 27.316
  21:03:45 STEP 2440 H=2440:00 +CPU= 16.746
  21:04:02 STEP 2441 H=2441:00 +CPU= 16.094
  21:04:30 STEP 2442 H=2442:00 +CPU= 27.140
  21:04:47 STEP 2443 H=2443:00 +CPU= 16.272
  21:05:06 STEP 2444 H=2444:00 +CPU= 17.584
  21:05:41 STEP 2445 H=2445:00 +CPU= 30.055
  21:06:03 STEP 2446 H=2446:00 +CPU= 18.880
  21:06:25 STEP 2447 H=2447:00 +CPU= 18.831
  21:07:08 STEP 2448 H=2448:00 +CPU= 36.421
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_101.zip
Uploading the intermediate file: upload_file_101.zip
  21:07:29 STEP 2449 H=2449:00 +CPU= 18.162
  21:07:49 STEP 2450 H=2450:00 +CPU= 17.930
  21:08:20 STEP 2451 H=2451:00 +CPU= 27.928
  21:08:40 STEP 2452 H=2452:00 +CPU= 17.841
  21:09:04 STEP 2453 H=2453:00 +CPU= 20.503
  21:09:45 STEP 2454 H=2454:00 +CPU= 33.717
  21:10:07 STEP 2455 H=2455:00 +CPU= 18.954
  21:10:31 STEP 2456 H=2456:00 +CPU= 20.417
  21:11:06 STEP 2457 H=2457:00 +CPU= 29.468
  21:11:29 STEP 2458 H=2458:00 +CPU= 19.506
  21:11:48 STEP 2459 H=2459:00 +CPU= 17.527
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.440] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4749MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.440] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.440] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.440] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4749 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     2        (1005) 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEB 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARCINB 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211205:1675735925:4181.447] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 LAITLI 
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211206:1675735926:4182.447] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:1841:1841] [20230206:211206:1675735926:4182.447] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000C1A8BD  laitli_                   145  laitli.F90
oifs_43r3_model.e  0000000000DBE1A4  larcinb_                  354  larcinb.F90
oifs_43r3_model.e  0000000000D7EBA8  lapineb_                  451  lapineb.F90
oifs_43r3_model.e  0000000000D7B935  call_sl_                  363  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0961
(argv4) batchid: 952
(argv5) wuid: 12169605
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_aca4d39a21aedda0c6a55730113f5c3b to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1983050100_123_952_12169605.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1983050100_123_952_12169605.zip
ic_ancil_file: ic_ancil_12169605
ifsdata_file: ifsdata_12169605
climate_data_file: clim_data_12169605
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6408ba3644ffb3bd7fae859a6242b274 to: /var/lib/boinc-client/slots/0/ic_ancil_12169605.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12169605.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2e81f408ce31654b5cc2aa9d1a7a0f63 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12169605.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12169605.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12169605.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12169605.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12169605.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12169605.xml
-- Model is restarting --
Adjusting last_iter, 2460, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12169605.xml
last_cpu_time: 49144
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245
The child process has been launched with process id: 2135
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:181000:1675811400:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:10:41 STEP 2448 H=2448:00 +CPU= 34.027
  18:10:56 STEP 2449 H=2449:00 +CPU= 15.078
  18:11:15 STEP 2450 H=2450:00 +CPU= 18.870
  18:11:39 STEP 2451 H=2451:00 +CPU= 23.290
  18:11:54 STEP 2452 H=2452:00 +CPU= 15.516
  18:12:09 STEP 2453 H=2453:00 +CPU= 14.992
  18:12:33 STEP 2454 H=2454:00 +CPU= 24.023
  18:12:49 STEP 2455 H=2455:00 +CPU= 15.415
  18:13:05 STEP 2456 H=2456:00 +CPU= 16.141
  18:13:30 STEP 2457 H=2457:00 +CPU= 24.073
  18:13:47 STEP 2458 H=2458:00 +CPU= 16.112
  18:14:03 STEP 2459 H=2459:00 +CPU= 15.473
  18:14:30 STEP 2460 H=2460:00 +CPU= 26.778
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002460
  18:14:46 STEP 2461 H=2461:00 +CPU= 15.749
  18:15:03 STEP 2462 H=2462:00 +CPU= 16.069
  18:15:27 STEP 2463 H=2463:00 +CPU= 24.136
  18:15:44 STEP 2464 H=2464:00 +CPU= 16.028
  18:16:00 STEP 2465 H=2465:00 +CPU= 15.680
  18:16:25 STEP 2466 H=2466:00 +CPU= 24.488
  18:16:41 STEP 2467 H=2467:00 +CPU= 15.573
  18:16:58 STEP 2468 H=2468:00 +CPU= 15.886
  18:17:22 STEP 2469 H=2469:00 +CPU= 24.161
  18:17:39 STEP 2470 H=2470:00 +CPU= 15.926
  18:17:55 STEP 2471 H=2471:00 +CPU= 15.980
  18:18:29 STEP 2472 H=2472:00 +CPU= 32.831
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_102.zip
Uploading the intermediate file: upload_file_102.zip
  18:18:52 STEP 2473 H=2473:00 +CPU= 21.177
  18:19:16 STEP 2474 H=2474:00 +CPU= 21.510
  18:19:51 STEP 2475 H=2475:00 +CPU= 33.288
  18:20:14 STEP 2476 H=2476:00 +CPU= 21.496
  18:20:36 STEP 2477 H=2477:00 +CPU= 20.593
  18:21:11 STEP 2478 H=2478:00 +CPU= 32.919
  18:21:32 STEP 2479 H=2479:00 +CPU= 20.392
  18:21:55 STEP 2480 H=2480:00 +CPU= 21.206
  18:22:27 STEP 2481 H=2481:00 +CPU= 30.446
  18:22:49 STEP 2482 H=2482:00 +CPU= 21.190
  18:23:12 STEP 2483 H=2483:00 +CPU= 21.066
  18:23:50 STEP 2484 H=2484:00 +CPU= 35.351
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002484
  18:24:11 STEP 2485 H=2485:00 +CPU= 20.086
  18:24:33 STEP 2486 H=2486:00 +CPU= 20.494
  18:25:06 STEP 2487 H=2487:00 +CPU= 30.853
  18:25:28 STEP 2488 H=2488:00 +CPU= 20.822
  18:25:48 STEP 2489 H=2489:00 +CPU= 19.372
  18:26:21 STEP 2490 H=2490:00 +CPU= 31.145
  18:26:42 STEP 2491 H=2491:00 +CPU= 19.279
  18:27:03 STEP 2492 H=2492:00 +CPU= 19.895
  18:27:35 STEP 2493 H=2493:00 +CPU= 30.357
  18:27:56 STEP 2494 H=2494:00 +CPU= 19.684
  18:28:18 STEP 2495 H=2495:00 +CPU= 21.165
  18:29:00 STEP 2496 H=2496:00 +CPU= 38.509
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_103.zip
Uploading the intermediate file: upload_file_103.zip
  18:29:21 STEP 2497 H=2497:00 +CPU= 19.725
  18:29:42 STEP 2498 H=2498:00 +CPU= 19.993
  18:30:15 STEP 2499 H=2499:00 +CPU= 31.004
  18:30:35 STEP 2500 H=2500:00 +CPU= 18.660
  18:30:55 STEP 2501 H=2501:00 +CPU= 18.379
  18:31:25 STEP 2502 H=2502:00 +CPU= 28.938
  18:31:43 STEP 2503 H=2503:00 +CPU= 17.008
  18:32:02 STEP 2504 H=2504:00 +CPU= 18.083
  18:32:31 STEP 2505 H=2505:00 +CPU= 27.710
  18:32:51 STEP 2506 H=2506:00 +CPU= 18.604
  18:33:10 STEP 2507 H=2507:00 +CPU= 18.368
  18:33:44 STEP 2508 H=2508:00 +CPU= 31.616
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002508
  18:34:05 STEP 2509 H=2509:00 +CPU= 19.995
  18:34:27 STEP 2510 H=2510:00 +CPU= 20.887
  18:34:55 STEP 2511 H=2511:00 +CPU= 26.312
  18:35:11 STEP 2512 H=2512:00 +CPU= 16.030
  18:35:28 STEP 2513 H=2513:00 +CPU= 15.801
  18:35:54 STEP 2514 H=2514:00 +CPU= 25.843
  18:36:15 STEP 2515 H=2515:00 +CPU= 19.699
  18:36:36 STEP 2516 H=2516:00 +CPU= 19.096
  18:37:06 STEP 2517 H=2517:00 +CPU= 28.559
  18:37:27 STEP 2518 H=2518:00 +CPU= 19.974
  18:37:48 STEP 2519 H=2519:00 +CPU= 19.403
  18:38:24 STEP 2520 H=2520:00 +CPU= 34.225
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245_104.zip
Uploading the intermediate file: upload_file_104.zip
  18:38:42 STEP 2521 H=2521:00 +CPU= 17.698
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183848:1675813128:1728.266] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183849:1675813129:1729.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:2135:2135] [20230207:183849:1675813129:1729.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000010227EC  cloudsc_                 1107  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F10C1B  callpar_                 1011  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0961
(argv4) batchid: 952
(argv5) wuid: 12169605
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169605
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_aca4d39a21aedda0c6a55730113f5c3b to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1983050100_123_952_12169605.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0961_1983050100_123_952_12169605.zip
ic_ancil_file: ic_ancil_12169605
ifsdata_file: ifsdata_12169605
climate_data_file: clim_data_12169605
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6408ba3644ffb3bd7fae859a6242b274 to: /var/lib/boinc-client/slots/0/ic_ancil_12169605.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12169605.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2e81f408ce31654b5cc2aa9d1a7a0f63 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12169605.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12169605.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12169605.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12169605.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12169605.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12169605.xml
-- Model is restarting --
Adjusting last_iter, 2521, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12169605.xml
last_cpu_time: 50766
upload_file_number: 105
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0961_1983050100_123_952_12169605_1_r515707245
The child process has been launched with process id: 1904
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1904:1904] [20230209:121609:1675962969:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:16:50 STEP 2520 H=2520:00 +CPU= 33.708
  12:17:05 STEP 2521 H=2521:00 +CPU= 15.121
  12:17:24 STEP 2522 H=2522:00 +CPU= 18.578
  12:17:48 STEP 2523 H=2523:00 +CPU= 23.687
  12:18:04 STEP 2524 H=2524:00 +CPU= 15.523
  12:18:19 STEP 2525 H=2525:00 +CPU= 15.144
  12:18:44 STEP 2526 H=2526:00 +CPU= 24.484
  12:19:02 STEP 2527 H=2527:00 +CPU= 17.404
  12:19:20 STEP 2528 H=2528:00 +CPU= 17.012
  12:19:47 STEP 2529 H=2529:00 +CPU= 25.936
  12:20:05 STEP 2530 H=2530:00 +CPU= 17.407
  12:20:24 STEP 2531 H=2531:00 +CPU= 17.470
  12:20:53 STEP 2532 H=2532:00 +CPU= 27.774
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002532
  12:21:12 STEP 2533 H=2533:00 +CPU= 17.483
  12:21:39 STEP 2534 H=2534:00 +CPU= 21.887
  12:22:09 STEP 2535 H=2535:00 +CPU= 27.622
  12:22:30 STEP 2536 H=2536:00 +CPU= 19.978
  12:22:54 STEP 2537 H=2537:00 +CPU= 20.598
  12:23:37 STEP 2538 H=2538:00 +CPU= 33.645
  12:24:03 STEP 2539 H=2539:00 +CPU= 21.369
  12:24:27 STEP 2540 H=2540:00 +CPU= 21.838
  12:25:04 STEP 2541 H=2541:00 +CPU= 32.914
  12:25:27 STEP 2542 H=2542:00 +CPU= 20.776
  12:25:49 STEP 2543 H=2543:00 +CPU= 20.692
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:48:26 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 7,948,800 43,577 0.0055
09 May 2023 16:48:24 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 6,739,200 36,844 0.0055
09 May 2023 16:48:23 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 5,875,200 32,432 0.0055
09 May 2023 16:48:22 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 5,011,200 28,021 0.0056
09 May 2023 16:48:20 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 4,147,200 23,610 0.0057
09 May 2023 16:48:18 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 3,283,200 19,189 0.0058
09 May 2023 16:48:16 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 2,160,000 12,509 0.0058
09 May 2023 16:47:45 1534946 22305357 oifs_43r3_ps_0961_1983050100_123_952_12169605_1 864,000 4,293 0.0050


©2024 cpdn.org