climateprediction.net home page
Task 22287294

Task 22287294

Name oifs_43r3_ps_0379_2015050100_123_984_12201023_0
Workunit 12201023
Created 24 Dec 2022, 12:38:11 UTC
Sent 19 Jan 2023, 12:06:23 UTC
Report deadline 20 Mar 2023, 12:06:23 UTC
Received 20 Jan 2023, 6:01:26 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1491267
Run time 15 hours 52 min 34 sec
CPU time 15 hours 32 min 28 sec
Validate state Invalid
Credit 2,298.24
Device peak FLOPS 4.56 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,574.83 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.12 MB
Stderr 
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
nal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407639:407639] [20230119:230658:1674169618:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:08:41 STEP  576 H= 576:00 +CPU= 89.676
  23:09:29 STEP  577 H= 577:00 +CPU= 47.584
  23:10:17 STEP  578 H= 578:00 +CPU= 46.179
  23:11:23 STEP  579 H= 579:00 +CPU= 65.189
  23:12:04 STEP  580 H= 580:00 +CPU= 40.628
  23:12:48 STEP  581 H= 581:00 +CPU= 43.354
  23:14:05 STEP  582 H= 582:00 +CPU= 72.064
  23:14:51 STEP  583 H= 583:00 +CPU= 45.819
  23:15:40 STEP  584 H= 584:00 +CPU= 48.927
  23:16:50 STEP  585 H= 585:00 +CPU= 68.421
  23:17:39 STEP  586 H= 586:00 +CPU= 48.489
  23:18:26 STEP  587 H= 587:00 +CPU= 47.378
  23:19:45 STEP  588 H= 588:00 +CPU= 78.090
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000588
  23:20:32 STEP  589 H= 589:00 +CPU= 46.700
  23:21:23 STEP  590 H= 590:00 +CPU= 48.942
  23:22:35 STEP  591 H= 591:00 +CPU= 70.484
  23:23:23 STEP  592 H= 592:00 +CPU= 48.334
  23:24:10 STEP  593 H= 593:00 +CPU= 46.451
  23:25:22 STEP  594 H= 594:00 +CPU= 71.482
  23:26:10 STEP  595 H= 595:00 +CPU= 46.986
  23:26:59 STEP  596 H= 596:00 +CPU= 48.177
  23:28:09 STEP  597 H= 597:00 +CPU= 70.083
  23:29:00 STEP  598 H= 598:00 +CPU= 50.663
  23:29:49 STEP  599 H= 599:00 +CPU= 48.036
  23:31:19 STEP  600 H= 600:00 +CPU= 84.983
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_24.zip
Uploading the intermediate file: upload_file_24.zip
  23:32:08 STEP  601 H= 601:00 +CPU= 48.452
  23:32:59 STEP  602 H= 602:00 +CPU= 49.468
  23:34:10 STEP  603 H= 603:00 +CPU= 70.173
  23:35:01 STEP  604 H= 604:00 +CPU= 50.553
  23:35:49 STEP  605 H= 605:00 +CPU= 47.260
  23:37:06 STEP  606 H= 606:00 +CPU= 73.736
  23:37:56 STEP  607 H= 607:00 +CPU= 49.638
  23:38:48 STEP  608 H= 608:00 +CPU= 50.670
  23:40:02 STEP  609 H= 609:00 +CPU= 73.001
  23:40:55 STEP  610 H= 610:00 +CPU= 52.619
  23:41:48 STEP  611 H= 611:00 +CPU= 51.931
  23:43:11 STEP  612 H= 612:00 +CPU= 81.474
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000612
  23:44:02 STEP  613 H= 613:00 +CPU= 49.991
  23:44:54 STEP  614 H= 614:00 +CPU= 50.732
  23:46:10 STEP  615 H= 615:00 +CPU= 75.383
  23:47:00 STEP  616 H= 616:00 +CPU= 49.748
  23:47:51 STEP  617 H= 617:00 +CPU= 50.034
  23:51:18 STEP  618 H= 618:00 +CPU= 78.729
  23:52:08 STEP  619 H= 619:00 +CPU= 47.952
  23:52:58 STEP  620 H= 620:00 +CPU= 48.888
  23:54:06 STEP  621 H= 621:00 +CPU= 67.719
  23:54:56 STEP  622 H= 622:00 +CPU= 48.944
  23:55:50 STEP  623 H= 623:00 +CPU= 52.162
  23:57:23 STEP  624 H= 624:00 +CPU= 85.067
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_25.zip
Uploading the intermediate file: upload_file_25.zip
  23:58:12 STEP  625 H= 625:00 +CPU= 48.646
  23:59:00 STEP  626 H= 626:00 +CPU= 47.591
  00:00:09 STEP  627 H= 627:00 +CPU= 68.729
  00:00:58 STEP  628 H= 628:00 +CPU= 48.460
  00:01:46 STEP  629 H= 629:00 +CPU= 47.439
  00:02:58 STEP  630 H= 630:00 +CPU= 71.072
  00:03:46 STEP  631 H= 631:00 +CPU= 47.763
  00:04:36 STEP  632 H= 632:00 +CPU= 49.327
  00:05:47 STEP  633 H= 633:00 +CPU= 70.971
  00:06:36 STEP  634 H= 634:00 +CPU= 48.566
  00:07:24 STEP  635 H= 635:00 +CPU= 47.438
  00:08:43 STEP  636 H= 636:00 +CPU= 78.281
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000636
  00:09:31 STEP  637 H= 637:00 +CPU= 47.332
  00:10:20 STEP  638 H= 638:00 +CPU= 48.591
  00:11:32 STEP  639 H= 639:00 +CPU= 71.004
  00:12:21 STEP  640 H= 640:00 +CPU= 48.479
  00:13:11 STEP  641 H= 641:00 +CPU= 49.067
  00:14:26 STEP  642 H= 642:00 +CPU= 72.958
  00:15:19 STEP  643 H= 643:00 +CPU= 52.614
  00:16:17 STEP  644 H= 644:00 +CPU= 57.089
  00:17:41 STEP  645 H= 645:00 +CPU= 82.289
  00:18:39 STEP  646 H= 646:00 +CPU= 57.111
  00:19:40 STEP  647 H= 647:00 +CPU= 59.737
  00:21:21 STEP  648 H= 648:00 +CPU= 97.670
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_26.zip
Uploading the intermediate file: upload_file_26.zip
  00:22:17 STEP  649 H= 649:00 +CPU= 55.620
  00:23:17 STEP  650 H= 650:00 +CPU= 58.128
  00:24:42 STEP  651 H= 651:00 +CPU= 84.232
  00:25:40 STEP  652 H= 652:00 +CPU= 56.959
  00:26:37 STEP  653 H= 653:00 +CPU= 56.190
  00:28:05 STEP  654 H= 654:00 +CPU= 86.365
  00:29:02 STEP  655 H= 655:00 +CPU= 55.951
  00:30:02 STEP  656 H= 656:00 +CPU= 58.635
  00:31:28 STEP  657 H= 657:00 +CPU= 84.848
  00:32:25 STEP  658 H= 658:00 +CPU= 55.548
  00:33:23 STEP  659 H= 659:00 +CPU= 57.001
  00:35:00 STEP  660 H= 660:00 +CPU= 95.927
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000660
  00:35:56 STEP  661 H= 661:00 +CPU= 54.851
  00:36:55 STEP  662 H= 662:00 +CPU= 58.139
  00:38:23 STEP  663 H= 663:00 +CPU= 86.327
  00:39:21 STEP  664 H= 664:00 +CPU= 57.285
  00:40:17 STEP  665 H= 665:00 +CPU= 55.195
  00:41:47 STEP  666 H= 666:00 +CPU= 88.229
  00:42:43 STEP  667 H= 667:00 +CPU= 55.087
  00:43:40 STEP  668 H= 668:00 +CPU= 56.266
  00:45:07 STEP  669 H= 669:00 +CPU= 85.709
  00:46:07 STEP  670 H= 670:00 +CPU= 58.632
  00:47:03 STEP  671 H= 671:00 +CPU= 54.511
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0379
(argv4) batchid: 984
(argv5) wuid: 12201023
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6b445eb8963d80a84813433411e4c097 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0379_2015050100_123_984_12201023.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0379_2015050100_123_984_12201023.zip
ic_ancil_file: ic_ancil_12201023
ifsdata_file: ifsdata_12201023
climate_data_file: clim_data_12201023
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c3a7a1d21fa2cbee2d96b97baf803571 to: /var/lib/boinc-client/slots/9/ic_ancil_12201023.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12201023.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b30d3e77d0a6ebf61b21e2da7c5280d to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12201023.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12201023.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12201023.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12201023.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12201023.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12201023.xml
-- Model is restarting --
Adjusting last_iter, 672, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12201023.xml
last_cpu_time: 41711
upload_file_number: 27
last_iter: 671
last_upload: 2332800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209
The child process has been launched with process id: 407893
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:407893:407893] [20230120:005011:1674175811:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:52:10 STEP  672 H= 672:00 +CPU= 93.746
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_27.zip
Uploading the intermediate file: upload_file_27.zip
  00:53:04 STEP  673 H= 673:00 +CPU= 44.647
  00:53:51 STEP  674 H= 674:00 +CPU= 46.713
  00:54:59 STEP  675 H= 675:00 +CPU= 67.680
  00:55:45 STEP  676 H= 676:00 +CPU= 45.054
  00:56:33 STEP  677 H= 677:00 +CPU= 46.155
  00:57:46 STEP  678 H= 678:00 +CPU= 70.888
  00:58:33 STEP  679 H= 679:00 +CPU= 45.981
  00:59:22 STEP  680 H= 680:00 +CPU= 48.380
  01:00:30 STEP  681 H= 681:00 +CPU= 66.958
  01:01:08 STEP  682 H= 682:00 +CPU= 38.454
  01:01:51 STEP  683 H= 683:00 +CPU= 41.888
  01:03:07 STEP  684 H= 684:00 +CPU= 74.463
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000684
  01:03:50 STEP  685 H= 685:00 +CPU= 42.682
  01:04:38 STEP  686 H= 686:00 +CPU= 47.630
  01:05:46 STEP  687 H= 687:00 +CPU= 67.303
  01:06:26 STEP  688 H= 688:00 +CPU= 40.000
  01:07:09 STEP  689 H= 689:00 +CPU= 42.717
  01:08:20 STEP  690 H= 690:00 +CPU= 69.838
  01:09:07 STEP  691 H= 691:00 +CPU= 47.087
  01:09:57 STEP  692 H= 692:00 +CPU= 49.460
  01:11:06 STEP  693 H= 693:00 +CPU= 67.805
  01:11:50 STEP  694 H= 694:00 +CPU= 44.219
  01:12:33 STEP  695 H= 695:00 +CPU= 42.490
  01:13:54 STEP  696 H= 696:00 +CPU= 76.859
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_28.zip
Uploading the intermediate file: upload_file_28.zip
  01:14:34 STEP  697 H= 697:00 +CPU= 39.543
  01:15:18 STEP  698 H= 698:00 +CPU= 43.932
  01:16:25 STEP  699 H= 699:00 +CPU= 66.323
  01:17:06 STEP  700 H= 700:00 +CPU= 40.515
  01:17:48 STEP  701 H= 701:00 +CPU= 41.365
  01:18:59 STEP  702 H= 702:00 +CPU= 69.616
  01:19:48 STEP  703 H= 703:00 +CPU= 48.501
  01:20:36 STEP  704 H= 704:00 +CPU= 47.794
  01:21:46 STEP  705 H= 705:00 +CPU= 69.679
  01:22:37 STEP  706 H= 706:00 +CPU= 50.168
  01:23:24 STEP  707 H= 707:00 +CPU= 46.948
  01:24:43 STEP  708 H= 708:00 +CPU= 78.166
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000708
  01:25:33 STEP  709 H= 709:00 +CPU= 48.733
  01:26:21 STEP  710 H= 710:00 +CPU= 48.128
  01:27:32 STEP  711 H= 711:00 +CPU= 69.938
  01:28:23 STEP  712 H= 712:00 +CPU= 50.380
  01:29:10 STEP  713 H= 713:00 +CPU= 46.628
  01:30:21 STEP  714 H= 714:00 +CPU= 70.372
  01:31:10 STEP  715 H= 715:00 +CPU= 48.588
  01:31:59 STEP  716 H= 716:00 +CPU= 48.903
  01:33:09 STEP  717 H= 717:00 +CPU= 68.786
  01:34:00 STEP  718 H= 718:00 +CPU= 50.860
  01:34:52 STEP  719 H= 719:00 +CPU= 50.880
  01:36:16 STEP  720 H= 720:00 +CPU= 83.232
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_29.zip
Uploading the intermediate file: upload_file_29.zip
  01:37:07 STEP  721 H= 721:00 +CPU= 49.746
  01:37:56 STEP  722 H= 722:00 +CPU= 49.021
  01:39:06 STEP  723 H= 723:00 +CPU= 69.535
  01:39:57 STEP  724 H= 724:00 +CPU= 50.311
  01:40:45 STEP  725 H= 725:00 +CPU= 47.585
  01:41:56 STEP  726 H= 726:00 +CPU= 70.257
  01:42:45 STEP  727 H= 727:00 +CPU= 48.721
  01:43:36 STEP  728 H= 728:00 +CPU= 50.075
  01:44:46 STEP  729 H= 729:00 +CPU= 69.359
  01:45:37 STEP  730 H= 730:00 +CPU= 50.203
  01:46:26 STEP  731 H= 731:00 +CPU= 49.016
  01:47:46 STEP  732 H= 732:00 +CPU= 79.115
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000732
  01:48:36 STEP  733 H= 733:00 +CPU= 49.305
  01:49:27 STEP  734 H= 734:00 +CPU= 50.044
  01:50:36 STEP  735 H= 735:00 +CPU= 68.693
  01:51:26 STEP  736 H= 736:00 +CPU= 49.681
  01:52:15 STEP  737 H= 737:00 +CPU= 47.904
  01:53:28 STEP  738 H= 738:00 +CPU= 72.297
  01:54:15 STEP  739 H= 739:00 +CPU= 47.371
  01:55:06 STEP  740 H= 740:00 +CPU= 49.644
  01:56:17 STEP  741 H= 741:00 +CPU= 71.144
  01:57:06 STEP  742 H= 742:00 +CPU= 48.145
  01:57:57 STEP  743 H= 743:00 +CPU= 50.821
  01:59:21 STEP  744 H= 744:00 +CPU= 82.454
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_30.zip
Uploading the intermediate file: upload_file_30.zip
  02:00:10 STEP  745 H= 745:00 +CPU= 48.331
  02:01:00 STEP  746 H= 746:00 +CPU= 49.010
  02:02:12 STEP  747 H= 747:00 +CPU= 71.318
  02:03:00 STEP  748 H= 748:00 +CPU= 48.273
  02:03:50 STEP  749 H= 749:00 +CPU= 48.712
  02:05:04 STEP  750 H= 750:00 +CPU= 73.439
  02:05:51 STEP  751 H= 751:00 +CPU= 47.353
  02:06:42 STEP  752 H= 752:00 +CPU= 49.609
  02:07:54 STEP  753 H= 753:00 +CPU= 71.990
  02:08:42 STEP  754 H= 754:00 +CPU= 47.565
  02:09:32 STEP  755 H= 755:00 +CPU= 49.662
  02:10:53 STEP  756 H= 756:00 +CPU= 79.982
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000756
  02:11:40 STEP  757 H= 757:00 +CPU= 46.319
  02:12:30 STEP  758 H= 758:00 +CPU= 49.746
  02:13:41 STEP  759 H= 759:00 +CPU= 70.280
  02:14:27 STEP  760 H= 760:00 +CPU= 45.237
  02:15:21 STEP  761 H= 761:00 +CPU= 52.641
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0379
(argv4) batchid: 984
(argv5) wuid: 12201023
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6b445eb8963d80a84813433411e4c097 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0379_2015050100_123_984_12201023.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0379_2015050100_123_984_12201023.zip
ic_ancil_file: ic_ancil_12201023
ifsdata_file: ifsdata_12201023
climate_data_file: clim_data_12201023
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c3a7a1d21fa2cbee2d96b97baf803571 to: /var/lib/boinc-client/slots/9/ic_ancil_12201023.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12201023.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b30d3e77d0a6ebf61b21e2da7c5280d to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12201023.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12201023.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12201023.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12201023.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12201023.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12201023.xml
-- Model is restarting --
Adjusting last_iter, 762, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12201023.xml
last_cpu_time: 46748
upload_file_number: 31
last_iter: 743
last_upload: 2678400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209
The child process has been launched with process id: 408098
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408098:408098] [20230120:022229:1674181349:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:24:12 STEP  744 H= 744:00 +CPU= 89.286
  02:25:00 STEP  745 H= 745:00 +CPU= 47.460
  02:25:50 STEP  746 H= 746:00 +CPU= 48.365
  02:27:02 STEP  747 H= 747:00 +CPU= 72.037
  02:27:50 STEP  748 H= 748:00 +CPU= 47.419
  02:28:37 STEP  749 H= 749:00 +CPU= 46.659
  02:29:52 STEP  750 H= 750:00 +CPU= 73.785
  02:30:38 STEP  751 H= 751:00 +CPU= 45.545
  02:31:27 STEP  752 H= 752:00 +CPU= 48.437
  02:32:39 STEP  753 H= 753:00 +CPU= 71.697
  02:33:26 STEP  754 H= 754:00 +CPU= 46.454
  02:34:14 STEP  755 H= 755:00 +CPU= 47.491
  02:35:34 STEP  756 H= 756:00 +CPU= 79.053
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000756
  02:36:19 STEP  757 H= 757:00 +CPU= 45.233
  02:37:09 STEP  758 H= 758:00 +CPU= 49.108
  02:38:20 STEP  759 H= 759:00 +CPU= 70.602
  02:39:07 STEP  760 H= 760:00 +CPU= 46.503
  02:39:55 STEP  761 H= 761:00 +CPU= 47.520
  02:41:09 STEP  762 H= 762:00 +CPU= 73.248
  02:41:56 STEP  763 H= 763:00 +CPU= 46.534
  02:42:45 STEP  764 H= 764:00 +CPU= 47.949
  02:43:57 STEP  765 H= 765:00 +CPU= 71.374
  02:44:46 STEP  766 H= 766:00 +CPU= 48.634
  02:45:34 STEP  767 H= 767:00 +CPU= 48.059
  02:46:59 STEP  768 H= 768:00 +CPU= 83.374
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_31.zip
Uploading the intermediate file: upload_file_31.zip
  02:47:47 STEP  769 H= 769:00 +CPU= 47.153
  02:48:34 STEP  770 H= 770:00 +CPU= 47.090
  02:49:46 STEP  771 H= 771:00 +CPU= 70.944
  02:50:37 STEP  772 H= 772:00 +CPU= 50.122
  02:51:23 STEP  773 H= 773:00 +CPU= 46.203
  02:52:37 STEP  774 H= 774:00 +CPU= 73.406
  02:53:28 STEP  775 H= 775:00 +CPU= 50.095
  02:54:15 STEP  776 H= 776:00 +CPU= 46.277
  02:55:28 STEP  777 H= 777:00 +CPU= 72.522
  02:56:20 STEP  778 H= 778:00 +CPU= 51.261
  02:57:06 STEP  779 H= 779:00 +CPU= 45.578
  02:58:27 STEP  780 H= 780:00 +CPU= 80.138
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000780
  02:59:16 STEP  781 H= 781:00 +CPU= 48.506
  03:00:02 STEP  782 H= 782:00 +CPU= 46.058
  03:01:15 STEP  783 H= 783:00 +CPU= 72.047
  03:02:05 STEP  784 H= 784:00 +CPU= 49.855
  03:02:51 STEP  785 H= 785:00 +CPU= 45.428
  03:04:07 STEP  786 H= 786:00 +CPU= 75.421
  03:04:57 STEP  787 H= 787:00 +CPU= 49.128
  03:05:44 STEP  788 H= 788:00 +CPU= 46.671
  03:06:58 STEP  789 H= 789:00 +CPU= 72.644
  03:07:48 STEP  790 H= 790:00 +CPU= 49.789
  03:08:36 STEP  791 H= 791:00 +CPU= 47.872
  03:10:02 STEP  792 H= 792:00 +CPU= 85.115
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_32.zip
Uploading the intermediate file: upload_file_32.zip
  03:10:51 STEP  793 H= 793:00 +CPU= 48.466
  03:11:38 STEP  794 H= 794:00 +CPU= 46.694
  03:12:52 STEP  795 H= 795:00 +CPU= 73.146
  03:13:44 STEP  796 H= 796:00 +CPU= 50.757
  03:14:29 STEP  797 H= 797:00 +CPU= 44.968
  03:15:43 STEP  798 H= 798:00 +CPU= 73.120
  03:16:30 STEP  799 H= 799:00 +CPU= 47.223
  03:17:17 STEP  800 H= 800:00 +CPU= 46.183
  03:18:30 STEP  801 H= 801:00 +CPU= 71.776
  03:19:19 STEP  802 H= 802:00 +CPU= 49.044
  03:20:05 STEP  803 H= 803:00 +CPU= 45.617
  03:21:24 STEP  804 H= 804:00 +CPU= 77.906
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000804
  03:22:13 STEP  805 H= 805:00 +CPU= 48.448
  03:23:00 STEP  806 H= 806:00 +CPU= 47.187
  03:24:11 STEP  807 H= 807:00 +CPU= 70.089
  03:25:00 STEP  808 H= 808:00 +CPU= 48.731
  03:25:49 STEP  809 H= 809:00 +CPU= 48.169
  03:27:02 STEP  810 H= 810:00 +CPU= 72.427
  03:27:51 STEP  811 H= 811:00 +CPU= 48.034
  03:28:41 STEP  812 H= 812:00 +CPU= 49.934
  03:29:52 STEP  813 H= 813:00 +CPU= 70.429
  03:30:43 STEP  814 H= 814:00 +CPU= 50.539
  03:31:35 STEP  815 H= 815:00 +CPU= 50.882
  03:32:58 STEP  816 H= 816:00 +CPU= 82.025
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_33.zip
Uploading the intermediate file: upload_file_33.zip
  03:33:47 STEP  817 H= 817:00 +CPU= 48.755
  03:34:36 STEP  818 H= 818:00 +CPU= 48.687
  03:35:48 STEP  819 H= 819:00 +CPU= 70.770
  03:36:38 STEP  820 H= 820:00 +CPU= 49.595
  03:37:26 STEP  821 H= 821:00 +CPU= 47.885
  03:38:38 STEP  822 H= 822:00 +CPU= 71.179
  03:39:26 STEP  823 H= 823:00 +CPU= 47.413
  03:40:16 STEP  824 H= 824:00 +CPU= 49.829
  03:41:27 STEP  825 H= 825:00 +CPU= 69.796
  03:42:15 STEP  826 H= 826:00 +CPU= 48.182
  03:43:03 STEP  827 H= 827:00 +CPU= 46.601
  03:44:22 STEP  828 H= 828:00 +CPU= 78.263
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000828
  03:45:09 STEP  829 H= 829:00 +CPU= 47.119
  03:45:59 STEP  830 H= 830:00 +CPU= 49.582
  03:47:10 STEP  831 H= 831:00 +CPU= 70.161
  03:47:59 STEP  832 H= 832:00 +CPU= 48.476
  03:48:48 STEP  833 H= 833:00 +CPU= 48.518
  03:50:02 STEP  834 H= 834:00 +CPU= 73.062
  03:50:51 STEP  835 H= 835:00 +CPU= 48.302
  03:51:41 STEP  836 H= 836:00 +CPU= 49.247
  03:52:52 STEP  837 H= 837:00 +CPU= 70.953
  03:53:42 STEP  838 H= 838:00 +CPU= 48.961
  03:54:32 STEP  839 H= 839:00 +CPU= 49.647
  03:55:57 STEP  840 H= 840:00 +CPU= 82.763
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_34.zip
Uploading the intermediate file: upload_file_34.zip
  03:56:44 STEP  841 H= 841:00 +CPU= 47.094
  03:57:34 STEP  842 H= 842:00 +CPU= 49.382
  03:58:45 STEP  843 H= 843:00 +CPU= 70.645
  03:59:34 STEP  844 H= 844:00 +CPU= 47.984
  04:00:23 STEP  845 H= 845:00 +CPU= 47.161
  04:01:33 STEP  846 H= 846:00 +CPU= 68.445
  04:02:19 STEP  847 H= 847:00 +CPU= 45.588
  04:03:10 STEP  848 H= 848:00 +CPU= 48.966
  04:04:20 STEP  849 H= 849:00 +CPU= 69.504
  04:05:07 STEP  850 H= 850:00 +CPU= 46.176
  04:05:56 STEP  851 H= 851:00 +CPU= 48.415
  04:07:15 STEP  852 H= 852:00 +CPU= 77.858
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000852
  04:07:59 STEP  853 H= 853:00 +CPU= 44.289
  04:08:51 STEP  854 H= 854:00 +CPU= 50.887
  04:10:03 STEP  855 H= 855:00 +CPU= 71.855
  04:10:48 STEP  856 H= 856:00 +CPU= 44.250
  04:11:36 STEP  857 H= 857:00 +CPU= 46.950
  04:12:53 STEP  858 H= 858:00 +CPU= 75.197
  04:13:36 STEP  859 H= 859:00 +CPU= 42.787
  04:14:23 STEP  860 H= 860:00 +CPU= 46.653
  04:15:34 STEP  861 H= 861:00 +CPU= 70.600
  04:16:21 STEP  862 H= 862:00 +CPU= 46.035
  04:17:09 STEP  863 H= 863:00 +CPU= 47.181
  04:18:46 STEP  864 H= 864:00 +CPU= 84.377
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_35.zip
Uploading the intermediate file: upload_file_35.zip
  04:19:35 STEP  865 H= 865:00 +CPU= 47.576
  04:20:25 STEP  866 H= 866:00 +CPU= 47.517
  04:21:38 STEP  867 H= 867:00 +CPU= 72.042
  04:22:26 STEP  868 H= 868:00 +CPU= 47.134
  04:23:09 STEP  869 H= 869:00 +CPU= 43.014
  04:24:24 STEP  870 H= 870:00 +CPU= 73.068
  04:25:11 STEP  871 H= 871:00 +CPU= 46.651
  04:26:03 STEP  872 H= 872:00 +CPU= 51.253
  04:27:32 STEP  873 H= 873:00 +CPU= 86.890
  04:28:30 STEP  874 H= 874:00 +CPU= 57.530
  04:29:25 STEP  875 H= 875:00 +CPU= 53.861
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0379
(argv4) batchid: 984
(argv5) wuid: 12201023
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6b445eb8963d80a84813433411e4c097 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0379_2015050100_123_984_12201023.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0379_2015050100_123_984_12201023.zip
ic_ancil_file: ic_ancil_12201023
ifsdata_file: ifsdata_12201023
climate_data_file: clim_data_12201023
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c3a7a1d21fa2cbee2d96b97baf803571 to: /var/lib/boinc-client/slots/9/ic_ancil_12201023.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12201023.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b30d3e77d0a6ebf61b21e2da7c5280d to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12201023.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12201023.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12201023.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12201023.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12201023.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12201023.xml
-- Model is restarting --
Adjusting last_iter, 876, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12201023.xml
last_cpu_time: 54300
upload_file_number: 36
last_iter: 863
last_upload: 3110400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209
The child process has been launched with process id: 408411
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Debian14:1:1:408411:408411] [20230120:043243:1674189163:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:34:27 STEP  864 H= 864:00 +CPU= 87.097
  04:35:19 STEP  865 H= 865:00 +CPU= 45.574
  04:36:01 STEP  866 H= 866:00 +CPU= 40.093
  04:37:19 STEP  867 H= 867:00 +CPU= 73.264
  04:38:08 STEP  868 H= 868:00 +CPU= 48.472
  04:38:57 STEP  869 H= 869:00 +CPU= 48.306
  04:40:11 STEP  870 H= 870:00 +CPU= 73.064
  04:41:01 STEP  871 H= 871:00 +CPU= 49.541
  04:41:53 STEP  872 H= 872:00 +CPU= 50.553
  04:43:05 STEP  873 H= 873:00 +CPU= 71.437
  04:43:57 STEP  874 H= 874:00 +CPU= 51.612
  04:44:51 STEP  875 H= 875:00 +CPU= 52.117
  04:46:15 STEP  876 H= 876:00 +CPU= 82.112
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000876
  04:47:03 STEP  877 H= 877:00 +CPU= 47.771
  04:47:52 STEP  878 H= 878:00 +CPU= 48.888
  04:49:06 STEP  879 H= 879:00 +CPU= 72.723
  04:49:58 STEP  880 H= 880:00 +CPU= 50.739
  04:50:47 STEP  881 H= 881:00 +CPU= 48.312
  04:52:04 STEP  882 H= 882:00 +CPU= 76.111
  04:52:55 STEP  883 H= 883:00 +CPU= 50.602
  04:53:45 STEP  884 H= 884:00 +CPU= 49.075
  04:54:53 STEP  885 H= 885:00 +CPU= 67.101
  04:55:47 STEP  886 H= 886:00 +CPU= 52.553
  04:56:40 STEP  887 H= 887:00 +CPU= 52.499
  04:58:09 STEP  888 H= 888:00 +CPU= 83.305
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201023/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2015050100_123_984_12201023_0_r547112209_36.zip
Uploading the intermediate file: upload_file_36.zip
  04:59:00 STEP  889 H= 889:00 +CPU= 50.005
  04:59:49 STEP  890 H= 890:00 +CPU= 48.731
  05:00:58 STEP  891 H= 891:00 +CPU= 67.844
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:29:23 1491267 22287294 oifs_43r3_ps_0379_2015050100_123_984_12201023_0 1,641,600 27,500 0.0168


©2024 climateprediction.net