<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:06:06 STEP 504 H= 504:00 +CPU= 42.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:11:09 STEP 505 H= 505:00 +CPU= 22.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:13:53 STEP 506 H= 506:00 +CPU= 22.248
11:14:27 STEP 507 H= 507:00 +CPU= 34.219
11:14:49 STEP 508 H= 508:00 +CPU= 21.904
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:22:52 STEP 509 H= 509:00 +CPU= 20.354
11:23:30 STEP 510 H= 510:00 +CPU= 37.213
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:27:05 STEP 511 H= 511:00 +CPU= 23.223
11:27:31 STEP 512 H= 512:00 +CPU= 24.876
11:28:08 STEP 513 H= 513:00 +CPU= 36.289
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:44:55 STEP 514 H= 514:00 +CPU= 24.312
11:45:20 STEP 515 H= 515:00 +CPU= 24.439
11:46:02 STEP 516 H= 516:00 +CPU= 41.873
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:47:44 STEP 517 H= 517:00 +CPU= 20.942
11:48:05 STEP 518 H= 518:00 +CPU= 21.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:49:50 STEP 519 H= 519:00 +CPU= 32.870
11:50:11 STEP 520 H= 520:00 +CPU= 21.306
11:50:32 STEP 521 H= 521:00 +CPU= 20.515
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:52:16 STEP 522 H= 522:00 +CPU= 33.491
11:52:36 STEP 523 H= 523:00 +CPU= 20.391
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:54:29 STEP 524 H= 524:00 +CPU= 20.854
11:55:03 STEP 525 H= 525:00 +CPU= 33.242
11:55:23 STEP 526 H= 526:00 +CPU= 20.739
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:56:36 STEP 527 H= 527:00 +CPU= 21.422
11:57:17 STEP 528 H= 528:00 +CPU= 37.719
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_21.zip
Uploading the intermediate file: upload_file_21.zip
11:57:36 STEP 529 H= 529:00 +CPU= 19.006
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:59:03 STEP 530 H= 530:00 +CPU= 21.320
11:59:35 STEP 531 H= 531:00 +CPU= 32.079
11:59:56 STEP 532 H= 532:00 +CPU= 20.751
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:01:27 STEP 533 H= 533:00 +CPU= 20.784
12:01:59 STEP 534 H= 534:00 +CPU= 32.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:03:31 STEP 535 H= 535:00 +CPU= 20.263
12:03:53 STEP 536 H= 536:00 +CPU= 22.113
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:05:37 STEP 537 H= 537:00 +CPU= 32.450
12:05:58 STEP 538 H= 538:00 +CPU= 21.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:07:19 STEP 539 H= 539:00 +CPU= 20.937
12:07:55 STEP 540 H= 540:00 +CPU= 35.547
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:09:16 STEP 541 H= 541:00 +CPU= 20.204
12:09:38 STEP 542 H= 542:00 +CPU= 22.069
12:10:10 STEP 543 H= 543:00 +CPU= 31.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:11:54 STEP 544 H= 544:00 +CPU= 21.573
12:12:15 STEP 545 H= 545:00 +CPU= 20.616
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:13:49 STEP 546 H= 546:00 +CPU= 33.968
12:14:09 STEP 547 H= 547:00 +CPU= 20.407
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:15:43 STEP 548 H= 548:00 +CPU= 21.374
12:16:15 STEP 549 H= 549:00 +CPU= 32.458
12:16:37 STEP 550 H= 550:00 +CPU= 21.019
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:18:18 STEP 551 H= 551:00 +CPU= 21.557
12:18:57 STEP 552 H= 552:00 +CPU= 38.184
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_22.zip
Uploading the intermediate file: upload_file_22.zip
12:19:16 STEP 553 H= 553:00 +CPU= 19.366
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:21:00 STEP 554 H= 554:00 +CPU= 20.736
12:21:33 STEP 555 H= 555:00 +CPU= 32.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:23:35 STEP 556 H= 556:00 +CPU= 21.065
12:23:57 STEP 557 H= 557:00 +CPU= 21.197
12:24:30 STEP 558 H= 558:00 +CPU= 33.381
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:26:22 STEP 559 H= 559:00 +CPU= 20.159
12:26:43 STEP 560 H= 560:00 +CPU= 21.254
12:27:19 STEP 561 H= 561:00 +CPU= 35.159
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:33:05 STEP 562 H= 562:00 +CPU= 24.678
12:33:29 STEP 563 H= 563:00 +CPU= 23.428
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:52:13 STEP 564 H= 564:00 +CPU= 40.271
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000564
12:52:36 STEP 565 H= 565:00 +CPU= 22.405
12:53:02 STEP 566 H= 566:00 +CPU= 25.283
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:00:43 STEP 567 H= 567:00 +CPU= 37.313
13:01:07 STEP 568 H= 568:00 +CPU= 23.448
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0089
(argv4) batchid: 958
(argv5) wuid: 12174733
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3b888d78cb29e0b788044e738156bcdf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip
ic_ancil_file: ic_ancil_12174733
ifsdata_file: ifsdata_12174733
climate_data_file: clim_data_12174733
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_41198eeba27a163b4104a5ac47617315 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12174733.xml
-- Model is restarting --
Adjusting last_iter, 569, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml
last_cpu_time: 16099
upload_file_number: 23
last_iter: 551
last_upload: 1987200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189
The child process has been launched with process id: 3234652
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:25:28 STEP 552 H= 552:00 +CPU= 40.885
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:34:30 STEP 553 H= 553:00 +CPU= 21.178
13:34:52 STEP 554 H= 554:00 +CPU= 21.808
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:36:58 STEP 555 H= 555:00 +CPU= 33.258
13:37:19 STEP 556 H= 556:00 +CPU= 21.058
13:37:40 STEP 557 H= 557:00 +CPU= 20.549
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:39:35 STEP 558 H= 558:00 +CPU= 34.194
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:41:07 STEP 559 H= 559:00 +CPU= 20.833
13:41:29 STEP 560 H= 560:00 +CPU= 21.121
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:43:23 STEP 561 H= 561:00 +CPU= 32.833
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:44:55 STEP 562 H= 562:00 +CPU= 20.557
13:45:17 STEP 563 H= 563:00 +CPU= 21.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:47:03 STEP 564 H= 564:00 +CPU= 35.814
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000564
13:48:36 STEP 565 H= 565:00 +CPU= 21.125
13:48:57 STEP 566 H= 566:00 +CPU= 21.029
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:50:51 STEP 567 H= 567:00 +CPU= 33.116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:52:14 STEP 568 H= 568:00 +CPU= 21.678
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:53:26 STEP 569 H= 569:00 +CPU= 20.977
13:53:59 STEP 570 H= 570:00 +CPU= 32.912
13:54:20 STEP 571 H= 571:00 +CPU= 20.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:56:01 STEP 572 H= 572:00 +CPU= 21.168
13:56:34 STEP 573 H= 573:00 +CPU= 32.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:58:36 STEP 574 H= 574:00 +CPU= 21.266
13:58:58 STEP 575 H= 575:00 +CPU= 21.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:00:48 STEP 576 H= 576:00 +CPU= 38.979
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_23.zip
Uploading the intermediate file: upload_file_23.zip
14:01:09 STEP 577 H= 577:00 +CPU= 20.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:03:03 STEP 578 H= 578:00 +CPU= 21.842
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:04:38 STEP 579 H= 579:00 +CPU= 33.009
14:04:59 STEP 580 H= 580:00 +CPU= 21.134
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:06:51 STEP 581 H= 581:00 +CPU= 20.301
14:07:25 STEP 582 H= 582:00 +CPU= 34.187
14:07:45 STEP 583 H= 583:00 +CPU= 20.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:10:20 STEP 584 H= 584:00 +CPU= 21.874
14:10:52 STEP 585 H= 585:00 +CPU= 32.419
14:11:13 STEP 586 H= 586:00 +CPU= 20.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:12:55 STEP 587 H= 587:00 +CPU= 21.014
14:13:31 STEP 588 H= 588:00 +CPU= 35.537
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:15:24 STEP 589 H= 589:00 +CPU= 21.137
14:15:45 STEP 590 H= 590:00 +CPU= 21.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:17:49 STEP 591 H= 591:00 +CPU= 32.838
14:18:10 STEP 592 H= 592:00 +CPU= 21.078
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:19:52 STEP 593 H= 593:00 +CPU= 20.347
14:20:25 STEP 594 H= 594:00 +CPU= 33.655
14:20:45 STEP 595 H= 595:00 +CPU= 19.786
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:22:37 STEP 596 H= 596:00 +CPU= 21.851
14:23:10 STEP 597 H= 597:00 +CPU= 32.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:25:23 STEP 598 H= 598:00 +CPU= 20.836
14:25:45 STEP 599 H= 599:00 +CPU= 21.442
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:27:46 STEP 600 H= 600:00 +CPU= 39.316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_24.zip
Uploading the intermediate file: upload_file_24.zip
14:28:06 STEP 601 H= 601:00 +CPU= 20.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:30:06 STEP 602 H= 602:00 +CPU= 21.329
14:30:39 STEP 603 H= 603:00 +CPU= 32.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:32:51 STEP 604 H= 604:00 +CPU= 21.344
14:33:13 STEP 605 H= 605:00 +CPU= 21.186
14:33:46 STEP 606 H= 606:00 +CPU= 33.151
14:34:06 STEP 607 H= 607:00 +CPU= 19.999
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:35:49 STEP 608 H= 608:00 +CPU= 21.747
14:36:22 STEP 609 H= 609:00 +CPU= 32.559
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:37:57 STEP 610 H= 610:00 +CPU= 22.606
14:38:18 STEP 611 H= 611:00 +CPU= 20.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:40:07 STEP 612 H= 612:00 +CPU= 36.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000612
14:42:19 STEP 613 H= 613:00 +CPU= 21.036
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:44:43 STEP 614 H= 614:00 +CPU= 21.616
14:45:17 STEP 615 H= 615:00 +CPU= 33.212
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:48:09 STEP 616 H= 616:00 +CPU= 21.845
14:48:32 STEP 617 H= 617:00 +CPU= 22.455
14:49:06 STEP 618 H= 618:00 +CPU= 33.591
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:50:48 STEP 619 H= 619:00 +CPU= 21.525
14:51:10 STEP 620 H= 620:00 +CPU= 21.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:53:04 STEP 621 H= 621:00 +CPU= 33.409
14:53:25 STEP 622 H= 622:00 +CPU= 20.755
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:55:08 STEP 623 H= 623:00 +CPU= 21.705
14:55:47 STEP 624 H= 624:00 +CPU= 38.405
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_25.zip
Uploading the intermediate file: upload_file_25.zip
14:58:01 STEP 625 H= 625:00 +CPU= 21.355
14:58:23 STEP 626 H= 626:00 +CPU= 22.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:00:18 STEP 627 H= 627:00 +CPU= 32.606
15:00:40 STEP 628 H= 628:00 +CPU= 21.442
15:01:00 STEP 629 H= 629:00 +CPU= 20.579
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:03:48 STEP 630 H= 630:00 +CPU= 33.416
15:04:09 STEP 631 H= 631:00 +CPU= 20.284
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:05:31 STEP 632 H= 632:00 +CPU= 21.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:06:44 STEP 633 H= 633:00 +CPU= 31.659
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:08:26 STEP 634 H= 634:00 +CPU= 21.337
15:08:47 STEP 635 H= 635:00 +CPU= 20.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:10:45 STEP 636 H= 636:00 +CPU= 36.234
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000636
15:11:06 STEP 637 H= 637:00 +CPU= 21.464
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:12:49 STEP 638 H= 638:00 +CPU= 21.409
15:13:21 STEP 639 H= 639:00 +CPU= 31.945
15:13:42 STEP 640 H= 640:00 +CPU= 20.957
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:15:34 STEP 641 H= 641:00 +CPU= 20.737
15:16:07 STEP 642 H= 642:00 +CPU= 33.250
15:16:27 STEP 643 H= 643:00 +CPU= 19.831
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:18:10 STEP 644 H= 644:00 +CPU= 21.417
15:18:42 STEP 645 H= 645:00 +CPU= 32.156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:20:24 STEP 646 H= 646:00 +CPU= 21.322
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0089
(argv4) batchid: 958
(argv5) wuid: 12174733
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3b888d78cb29e0b788044e738156bcdf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip
ic_ancil_file: ic_ancil_12174733
ifsdata_file: ifsdata_12174733
climate_data_file: clim_data_12174733
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_41198eeba27a163b4104a5ac47617315 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12174733.xml
-- Model is restarting --
Adjusting last_iter, 647, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml
last_cpu_time: 18531
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189
The child process has been launched with process id: 3254598
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
15:23:00 STEP 624 H= 624:00 +CPU= 37.158
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:24:52 STEP 625 H= 625:00 +CPU= 20.875
15:25:13 STEP 626 H= 626:00 +CPU= 21.054
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:27:19 STEP 627 H= 627:00 +CPU= 32.884
15:27:40 STEP 628 H= 628:00 +CPU= 21.127
15:28:00 STEP 629 H= 629:00 +CPU= 19.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:30:04 STEP 630 H= 630:00 +CPU= 33.197
15:30:24 STEP 631 H= 631:00 +CPU= 19.818
15:30:45 STEP 632 H= 632:00 +CPU= 20.937
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:32:59 STEP 633 H= 633:00 +CPU= 32.890
15:33:20 STEP 634 H= 634:00 +CPU= 20.925
15:33:41 STEP 635 H= 635:00 +CPU= 20.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:35:48 STEP 636 H= 636:00 +CPU= 35.453
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000636
15:36:08 STEP 637 H= 637:00 +CPU= 19.985
15:36:29 STEP 638 H= 638:00 +CPU= 20.698
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:38:33 STEP 639 H= 639:00 +CPU= 33.256
15:38:53 STEP 640 H= 640:00 +CPU= 20.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:40:36 STEP 641 H= 641:00 +CPU= 21.338
15:41:09 STEP 642 H= 642:00 +CPU= 33.192
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:43:20 STEP 643 H= 643:00 +CPU= 20.218
15:43:42 STEP 644 H= 644:00 +CPU= 21.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:45:16 STEP 645 H= 645:00 +CPU= 32.636
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:46:40 STEP 646 H= 646:00 +CPU= 21.231
15:47:01 STEP 647 H= 647:00 +CPU= 21.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:49:01 STEP 648 H= 648:00 +CPU= 38.011
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_26.zip
Uploading the intermediate file: upload_file_26.zip
15:49:21 STEP 649 H= 649:00 +CPU= 19.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:50:57 STEP 650 H= 650:00 +CPU= 21.510
15:51:30 STEP 651 H= 651:00 +CPU= 32.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:53:12 STEP 652 H= 652:00 +CPU= 21.267
15:53:34 STEP 653 H= 653:00 +CPU= 21.173
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:01:21 STEP 654 H= 654:00 +CPU= 34.388
16:01:43 STEP 655 H= 655:00 +CPU= 21.727
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:12:38 STEP 656 H= 656:00 +CPU= 22.479
16:13:13 STEP 657 H= 657:00 +CPU= 34.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:17:27 STEP 658 H= 658:00 +CPU= 22.305
16:17:51 STEP 659 H= 659:00 +CPU= 23.670
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:23:53 STEP 660 H= 660:00 +CPU= 39.110
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000660
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:57 STEP 661 H= 661:00 +CPU= 22.216
16:29:19 STEP 662 H= 662:00 +CPU= 22.529
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:36:55 STEP 663 H= 663:00 +CPU= 34.034
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:42:48 STEP 664 H= 664:00 +CPU= 21.836
16:43:10 STEP 665 H= 665:00 +CPU= 21.722
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:47:27 STEP 666 H= 666:00 +CPU= 35.396
16:47:50 STEP 667 H= 667:00 +CPU= 21.887
16:48:12 STEP 668 H= 668:00 +CPU= 22.324
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:55:00 STEP 669 H= 669:00 +CPU= 34.774
16:55:22 STEP 670 H= 670:00 +CPU= 22.478
16:55:45 STEP 671 H= 671:00 +CPU= 22.309
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:05:59 STEP 672 H= 672:00 +CPU= 40.911
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_27.zip
Uploading the intermediate file: upload_file_27.zip
17:06:21 STEP 673 H= 673:00 +CPU= 21.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:12:46 STEP 674 H= 674:00 +CPU= 22.679
17:13:22 STEP 675 H= 675:00 +CPU= 35.381
17:13:45 STEP 676 H= 676:00 +CPU= 22.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:17:19 STEP 677 H= 677:00 +CPU= 22.980
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:23:36 STEP 678 H= 678:00 +CPU= 35.501
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0089
(argv4) batchid: 958
(argv5) wuid: 12174733
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3b888d78cb29e0b788044e738156bcdf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip
ic_ancil_file: ic_ancil_12174733
ifsdata_file: ifsdata_12174733
climate_data_file: clim_data_12174733
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_41198eeba27a163b4104a5ac47617315 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12174733.xml
-- Model is restarting --
Adjusting last_iter, 678, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml
last_cpu_time: 19971
upload_file_number: 28
last_iter: 671
last_upload: 2419200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189
The child process has been launched with process id: 3273887
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:33:09 STEP 672 H= 672:00 +CPU= 40.062
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:36:04 STEP 673 H= 673:00 +CPU= 23.211
17:36:27 STEP 674 H= 674:00 +CPU= 22.815
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:40:45 STEP 675 H= 675:00 +CPU= 35.538
17:41:09 STEP 676 H= 676:00 +CPU= 23.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:47:13 STEP 677 H= 677:00 +CPU= 22.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:51:23 STEP 678 H= 678:00 +CPU= 36.801
17:51:46 STEP 679 H= 679:00 +CPU= 22.950
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:56:00 STEP 680 H= 680:00 +CPU= 22.684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:58:58 STEP 681 H= 681:00 +CPU= 35.467
17:59:21 STEP 682 H= 682:00 +CPU= 22.874
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:02:45 STEP 683 H= 683:00 +CPU= 22.788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:07:27 STEP 684 H= 684:00 +CPU= 38.569
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000684
18:07:51 STEP 685 H= 685:00 +CPU= 23.959
18:08:14 STEP 686 H= 686:00 +CPU= 22.537
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:11:51 STEP 687 H= 687:00 +CPU= 35.341
18:12:13 STEP 688 H= 688:00 +CPU= 22.312
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:18:07 STEP 689 H= 689:00 +CPU= 22.088
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:21:44 STEP 690 H= 690:00 +CPU= 35.871
18:22:08 STEP 691 H= 691:00 +CPU= 23.094
18:22:31 STEP 692 H= 692:00 +CPU= 22.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:25:49 STEP 693 H= 693:00 +CPU= 35.582
18:26:12 STEP 694 H= 694:00 +CPU= 23.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:30:18 STEP 695 H= 695:00 +CPU= 23.479
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
