Name | oifs_43r3_ps_0709_1988050100_123_957_12174353_0 |
Workunit | 12174353 |
Created | 21 Dec 2022, 14:37:19 UTC |
Sent | 18 Jan 2023, 3:22:05 UTC |
Report deadline | 19 Mar 2023, 3:22:05 UTC |
Received | 4 Feb 2023, 6:32:33 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1536107 |
Run time | 14 hours 15 min 15 sec |
CPU time | 13 hours 14 min 40 sec |
Validate state | Invalid |
Credit | 2,056.32 |
Device peak FLOPS | 1.00 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,409.44 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 3,106.63 MB |
Stderr | <core_client_version>7.20.2</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> :00 +CPU= 41.809 05:38:37 STEP 666 H= 666:00 +CPU= 67.556 05:39:19 STEP 667 H= 667:00 +CPU= 41.611 05:40:02 STEP 668 H= 668:00 +CPU= 42.954 05:41:08 STEP 669 H= 669:00 +CPU= 65.946 05:41:51 STEP 670 H= 670:00 +CPU= 42.669 05:42:35 STEP 671 H= 671:00 +CPU= 43.117 05:43:54 STEP 672 H= 672:00 +CPU= 77.917 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000660 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_27.zip Uploading the intermediate file: upload_file_27.zip 05:44:36 STEP 673 H= 673:00 +CPU= 42.167 05:45:19 STEP 674 H= 674:00 +CPU= 42.903 05:46:26 STEP 675 H= 675:00 +CPU= 66.122 05:47:09 STEP 676 H= 676:00 +CPU= 42.682 05:47:51 STEP 677 H= 677:00 +CPU= 41.630 05:48:59 STEP 678 H= 678:00 +CPU= 67.307 05:49:41 STEP 679 H= 679:00 +CPU= 41.715 05:50:24 STEP 680 H= 680:00 +CPU= 42.928 05:51:31 STEP 681 H= 681:00 +CPU= 66.376 05:52:14 STEP 682 H= 682:00 +CPU= 42.830 05:52:56 STEP 683 H= 683:00 +CPU= 41.558 05:54:10 STEP 684 H= 684:00 +CPU= 73.345 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000684 05:54:52 STEP 685 H= 685:00 +CPU= 41.887 05:55:35 STEP 686 H= 686:00 +CPU= 42.825 05:56:42 STEP 687 H= 687:00 +CPU= 66.512 05:57:25 STEP 688 H= 688:00 +CPU= 43.090 05:58:08 STEP 689 H= 689:00 +CPU= 41.889 05:59:16 STEP 690 H= 690:00 +CPU= 67.565 05:59:58 STEP 691 H= 691:00 +CPU= 41.633 06:00:41 STEP 692 H= 692:00 +CPU= 42.943 06:01:48 STEP 693 H= 693:00 +CPU= 66.305 06:02:31 STEP 694 H= 694:00 +CPU= 42.894 06:03:14 STEP 695 H= 695:00 +CPU= 42.910 06:04:32 STEP 696 H= 696:00 +CPU= 77.822 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000696 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000696 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000684 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_28.zip Uploading the intermediate file: upload_file_28.zip 06:05:15 STEP 697 H= 697:00 +CPU= 41.999 06:05:58 STEP 698 H= 698:00 +CPU= 43.212 06:07:04 STEP 699 H= 699:00 +CPU= 65.868 06:07:48 STEP 700 H= 700:00 +CPU= 42.816 06:08:29 STEP 701 H= 701:00 +CPU= 41.548 06:09:37 STEP 702 H= 702:00 +CPU= 67.561 06:10:20 STEP 703 H= 703:00 +CPU= 41.718 06:11:03 STEP 704 H= 704:00 +CPU= 42.920 06:12:09 STEP 705 H= 705:00 +CPU= 65.925 06:12:53 STEP 706 H= 706:00 +CPU= 42.994 06:13:34 STEP 707 H= 707:00 +CPU= 41.592 06:14:48 STEP 708 H= 708:00 +CPU= 73.177 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000708 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000708 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000708 06:15:30 STEP 709 H= 709:00 +CPU= 41.768 06:16:13 STEP 710 H= 710:00 +CPU= 42.943 06:17:20 STEP 711 H= 711:00 +CPU= 66.284 06:18:03 STEP 712 H= 712:00 +CPU= 42.755 06:18:45 STEP 713 H= 713:00 +CPU= 41.722 06:19:54 STEP 714 H= 714:00 +CPU= 68.025 06:20:36 STEP 715 H= 715:00 +CPU= 41.977 06:21:19 STEP 716 H= 716:00 +CPU= 42.645 06:22:25 STEP 717 H= 717:00 +CPU= 66.267 06:23:09 STEP 718 H= 718:00 +CPU= 42.782 06:23:52 STEP 719 H= 719:00 +CPU= 43.091 06:25:13 STEP 720 H= 720:00 +CPU= 79.290 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000720 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000720 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000708 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_29.zip Uploading the intermediate file: upload_file_29.zip 06:25:56 STEP 721 H= 721:00 +CPU= 42.249 06:26:39 STEP 722 H= 722:00 +CPU= 42.950 06:27:46 STEP 723 H= 723:00 +CPU= 66.424 06:28:29 STEP 724 H= 724:00 +CPU= 42.822 06:29:11 STEP 725 H= 725:00 +CPU= 41.751 06:30:19 STEP 726 H= 726:00 +CPU= 67.843 06:31:02 STEP 727 H= 727:00 +CPU= 41.802 06:31:45 STEP 728 H= 728:00 +CPU= 43.075 06:32:52 STEP 729 H= 729:00 +CPU= 66.590 06:33:35 STEP 730 H= 730:00 +CPU= 42.886 06:34:17 STEP 731 H= 731:00 +CPU= 41.744 06:35:32 STEP 732 H= 732:00 +CPU= 73.521 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000732 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000732 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000732 06:36:14 STEP 733 H= 733:00 +CPU= 42.044 06:36:57 STEP 734 H= 734:00 +CPU= 42.927 06:38:04 STEP 735 H= 735:00 +CPU= 66.492 06:38:47 STEP 736 H= 736:00 +CPU= 42.879 06:39:29 STEP 737 H= 737:00 +CPU= 41.820 06:40:37 STEP 738 H= 738:00 +CPU= 67.465 06:41:19 STEP 739 H= 739:00 +CPU= 41.663 06:42:03 STEP 740 H= 740:00 +CPU= 42.871 06:43:09 STEP 741 H= 741:00 +CPU= 66.140 06:43:53 STEP 742 H= 742:00 +CPU= 42.741 06:44:36 STEP 743 H= 743:00 +CPU= 43.069 06:45:55 STEP 744 H= 744:00 +CPU= 77.960 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000744 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000744 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000732 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_30.zip Uploading the intermediate file: upload_file_30.zip 06:46:38 STEP 745 H= 745:00 +CPU= 42.524 06:47:21 STEP 746 H= 746:00 +CPU= 42.685 06:48:32 STEP 747 H= 747:00 +CPU= 69.929 06:49:16 STEP 748 H= 748:00 +CPU= 43.764 06:49:58 STEP 749 H= 749:00 +CPU= 41.768 06:51:08 STEP 750 H= 750:00 +CPU= 69.539 06:51:50 STEP 751 H= 751:00 +CPU= 41.509 06:52:33 STEP 752 H= 752:00 +CPU= 42.924 06:53:41 STEP 753 H= 753:00 +CPU= 66.671 06:54:24 STEP 754 H= 754:00 +CPU= 42.912 06:55:06 STEP 755 H= 755:00 +CPU= 41.781 06:56:21 STEP 756 H= 756:00 +CPU= 74.024 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000756 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000756 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000756 06:57:02 STEP 757 H= 757:00 +CPU= 41.629 06:57:46 STEP 758 H= 758:00 +CPU= 43.489 06:58:53 STEP 759 H= 759:00 +CPU= 66.387 06:59:37 STEP 760 H= 760:00 +CPU= 42.909 07:00:19 STEP 761 H= 761:00 +CPU= 41.816 07:01:27 STEP 762 H= 762:00 +CPU= 68.159 07:02:09 STEP 763 H= 763:00 +CPU= 41.748 07:02:52 STEP 764 H= 764:00 +CPU= 42.697 07:04:00 STEP 765 H= 765:00 +CPU= 67.086 07:04:43 STEP 766 H= 766:00 +CPU= 42.975 07:05:27 STEP 767 H= 767:00 +CPU= 43.250 07:06:46 STEP 768 H= 768:00 +CPU= 78.378 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000768 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000768 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000756 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_31.zip Uploading the intermediate file: upload_file_31.zip 07:07:29 STEP 769 H= 769:00 +CPU= 42.496 07:08:13 STEP 770 H= 770:00 +CPU= 43.439 07:09:20 STEP 771 H= 771:00 +CPU= 66.746 07:10:03 STEP 772 H= 772:00 +CPU= 42.779 07:10:45 STEP 773 H= 773:00 +CPU= 41.713 07:11:54 STEP 774 H= 774:00 +CPU= 67.906 07:12:36 STEP 775 H= 775:00 +CPU= 41.789 07:13:19 STEP 776 H= 776:00 +CPU= 42.942 07:14:26 STEP 777 H= 777:00 +CPU= 66.839 07:15:09 STEP 778 H= 778:00 +CPU= 42.730 07:15:51 STEP 779 H= 779:00 +CPU= 41.650 07:17:06 STEP 780 H= 780:00 +CPU= 74.395 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780 07:17:48 STEP 781 H= 781:00 +CPU= 41.794 07:18:32 STEP 782 H= 782:00 +CPU= 42.969 07:19:39 STEP 783 H= 783:00 +CPU= 66.756 07:20:22 STEP 784 H= 784:00 +CPU= 42.844 07:21:04 STEP 785 H= 785:00 +CPU= 41.448 07:22:12 STEP 786 H= 786:00 +CPU= 67.949 07:22:54 STEP 787 H= 787:00 +CPU= 41.463 07:23:37 STEP 788 H= 788:00 +CPU= 42.911 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0709 (argv4) batchid: 957 (argv5) wuid: 12174353 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip ic_ancil_file: ic_ancil_12174353 ifsdata_file: ifsdata_12174353 climate_data_file: clim_data_12174353 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml -- Model is restarting -- Adjusting last_iter, 789, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml last_cpu_time: 45219 upload_file_number: 32 last_iter: 767 last_upload: 2764800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076 The child process has been launched with process id: 3821 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:3821:3821] [20230203:211112:1675483872:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:14:37 STEP 768 H= 768:00 +CPU= 88.368 21:15:28 STEP 769 H= 769:00 +CPU= 47.193 21:16:20 STEP 770 H= 770:00 +CPU= 49.599 21:17:49 STEP 771 H= 771:00 +CPU= 82.383 21:18:45 STEP 772 H= 772:00 +CPU= 53.045 21:19:41 STEP 773 H= 773:00 +CPU= 51.616 21:21:02 STEP 774 H= 774:00 +CPU= 77.692 21:21:51 STEP 775 H= 775:00 +CPU= 47.110 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0709 (argv4) batchid: 957 (argv5) wuid: 12174353 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip ic_ancil_file: ic_ancil_12174353 ifsdata_file: ifsdata_12174353 climate_data_file: clim_data_12174353 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml -- Model is restarting -- Adjusting last_iter, 775, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml last_cpu_time: 45716 upload_file_number: 32 last_iter: 767 last_upload: 2764800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076 The child process has been launched with process id: 10080 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:10080:10080] [20230203:212807:1675484887:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:33:48 STEP 768 H= 768:00 +CPU=101.445 21:34:44 STEP 769 H= 769:00 +CPU= 50.981 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0709 (argv4) batchid: 957 (argv5) wuid: 12174353 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip ic_ancil_file: ic_ancil_12174353 ifsdata_file: ifsdata_12174353 climate_data_file: clim_data_12174353 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml -- Model is restarting -- Adjusting last_iter, 770, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml last_cpu_time: 45899 upload_file_number: 32 last_iter: 767 last_upload: 2764800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076 The child process has been launched with process id: 12652 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:12652:12652] [20230203:214017:1675485617:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 21:46:33 STEP 768 H= 768:00 +CPU= 86.179 21:47:26 STEP 769 H= 769:00 +CPU= 50.444 21:48:30 STEP 770 H= 770:00 +CPU= 56.912 21:50:04 STEP 771 H= 771:00 +CPU= 85.648 21:51:01 STEP 772 H= 772:00 +CPU= 53.519 21:51:58 STEP 773 H= 773:00 +CPU= 52.139 21:53:38 STEP 774 H= 774:00 +CPU= 86.410 21:54:28 STEP 775 H= 775:00 +CPU= 47.321 21:55:20 STEP 776 H= 776:00 +CPU= 49.770 21:56:41 STEP 777 H= 777:00 +CPU= 78.374 21:57:35 STEP 778 H= 778:00 +CPU= 51.193 21:58:26 STEP 779 H= 779:00 +CPU= 49.206 21:59:56 STEP 780 H= 780:00 +CPU= 86.062 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780 22:01:00 STEP 781 H= 781:00 +CPU= 55.133 22:02:10 STEP 782 H= 782:00 +CPU= 58.902 22:04:09 STEP 783 H= 783:00 +CPU= 95.491 22:05:44 STEP 784 H= 784:00 +CPU= 69.703 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:07:31 STEP 785 H= 785:00 +CPU= 68.151 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:10:19 STEP 786 H= 786:00 +CPU=108.343 22:11:47 STEP 787 H= 787:00 +CPU= 66.332 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:14:00 STEP 788 H= 788:00 +CPU= 69.271 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:16:56 STEP 789 H= 789:00 +CPU=103.802 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:18:32 STEP 790 H= 790:00 +CPU= 65.144 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:21:08 STEP 791 H= 791:00 +CPU= 70.410 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1988050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0709 (argv4) batchid: 957 (argv5) wuid: 12174353 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_2c4a5711b467be9b2f09c8cd48f183f8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0709_1988050100_123_957_12174353.zip ic_ancil_file: ic_ancil_12174353 ifsdata_file: ifsdata_12174353 climate_data_file: clim_data_12174353 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b69ac54b4c383fc2d0e4cb362451deb9 to: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12174353.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fe3bb6a416b9412672bba12a96220ffa to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12174353.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12174353.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12174353.xml -- Model is restarting -- Adjusting last_iter, 792, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12174353.xml last_cpu_time: 47568 upload_file_number: 32 last_iter: 791 last_upload: 2764800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076 The child process has been launched with process id: 13899 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.013] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:222415:1675488255:0.014] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMGGhq0f+000780 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMSHhq0f+000780 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174353/ICMUAhq0f+000780 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0709_1988050100_123_957_12174353_0_r803898076_32.zip Uploading the intermediate file: upload_file_32.zip Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process ABORT! 1 !! *** WAVE MODEL HAS ABORTED [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.953] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3519 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.980] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223041:1675488641:385.983] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1 Process 1 thread 1 calling linux_trbk from intel_trbk() [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' : sigcontextptr=0x7fff1deb87a0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s) SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3519MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x800000364b], nsigs = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223044:1675488644:389.265] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223045:1675488645:390.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1 [EC_DRHOOK:Shaynovo:1:1:13899:13899] [20230203:223045:1675488645:390.266] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1 forrtl: error (76): Abort trap signal Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC903B Unknown Unknown Unknown oifs_43r3_model.e 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90 oifs_43r3_model.e 000000000081D8AF abor1_ 41 abor1.F90 oifs_43r3_model.e 00000000015F8ACA mpabort_ 50 mpabort.F oifs_43r3_model.e 00000000015D51C8 abort1_ 44 abort1.F oifs_43r3_model.e 000000000166B07F userin_ 553 userin.F oifs_43r3_model.e 00000000015D7FF0 initmdl_ 452 initmdl.F oifs_43r3_model.e 00000000008711E0 wavemdl_ 393 wavemdl.F oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90 oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown The child process terminated with status: 0 ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:44:04 | 1536107 | 22258991 | oifs_43r3_ps_0709_1988050100_123_957_12174353_0 | 1,468,800 | 22,673 | 0.0154 |
©2024 cpdn.org