climateprediction.net home page
Task 22269101

Task 22269101

Name oifs_43r3_ps_0258_1998050100_123_967_12183902_0
Workunit 12183902
Created 22 Dec 2022, 15:44:09 UTC
Sent 27 Dec 2022, 7:10:51 UTC
Report deadline 26 Jan 2023, 7:10:51 UTC
Received 31 Dec 2022, 20:38:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1533721
Run time 8 hours 52 min 21 sec
CPU time 7 hours 1 min 10 sec
Validate state Invalid
Credit 7,862.40
Device peak FLOPS 7.61 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,629.85 MB
Peak swap size 4,978.11 MB
Peak disk usage 7,237.50 MB
Stderr 
<core_client_version>7.20.4</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
he zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_68.zip
Uploading the intermediate file: upload_file_68.zip
  22:18:08 STEP 1657 H=1657:00 +CPU= 10.603
  22:18:22 STEP 1658 H=1658:00 +CPU= 11.835
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:18:50 STEP 1659 H=1659:00 +CPU= 16.568
  22:19:01 STEP 1660 H=1660:00 +CPU= 10.576
  22:19:12 STEP 1661 H=1661:00 +CPU= 10.762
  22:19:29 STEP 1662 H=1662:00 +CPU= 16.850
  22:19:39 STEP 1663 H=1663:00 +CPU= 10.478
  22:19:50 STEP 1664 H=1664:00 +CPU= 11.241
  22:20:07 STEP 1665 H=1665:00 +CPU= 16.982
  22:20:18 STEP 1666 H=1666:00 +CPU= 10.941
  22:20:29 STEP 1667 H=1667:00 +CPU= 10.518
  22:20:49 STEP 1668 H=1668:00 +CPU= 20.162
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001668
  22:21:00 STEP 1669 H=1669:00 +CPU= 10.751
  22:21:11 STEP 1670 H=1670:00 +CPU= 11.021
  22:21:28 STEP 1671 H=1671:00 +CPU= 17.011
  22:21:39 STEP 1672 H=1672:00 +CPU= 11.006
  22:21:50 STEP 1673 H=1673:00 +CPU= 10.525
  22:22:07 STEP 1674 H=1674:00 +CPU= 17.263
  22:22:18 STEP 1675 H=1675:00 +CPU= 10.768
  22:22:29 STEP 1676 H=1676:00 +CPU= 11.330
  22:22:46 STEP 1677 H=1677:00 +CPU= 16.841
  22:22:57 STEP 1678 H=1678:00 +CPU= 10.996
  22:23:11 STEP 1679 H=1679:00 +CPU= 11.678
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:23:44 STEP 1680 H=1680:00 +CPU= 19.764
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_69.zip
  22:23:54 STEP 1681 H=1681:00 +CPU= 10.146
Uploading the intermediate file: upload_file_69.zip
  22:24:05 STEP 1682 H=1682:00 +CPU= 11.063
  22:24:22 STEP 1683 H=1683:00 +CPU= 16.395
  22:24:33 STEP 1684 H=1684:00 +CPU= 10.836
  22:24:43 STEP 1685 H=1685:00 +CPU= 10.788
  22:25:01 STEP 1686 H=1686:00 +CPU= 18.010
  22:25:12 STEP 1687 H=1687:00 +CPU= 10.542
  22:25:23 STEP 1688 H=1688:00 +CPU= 11.241
  22:25:40 STEP 1689 H=1689:00 +CPU= 16.781
  22:25:51 STEP 1690 H=1690:00 +CPU= 10.926
  22:26:02 STEP 1691 H=1691:00 +CPU= 10.460
  22:26:20 STEP 1692 H=1692:00 +CPU= 18.313
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001692
  22:26:30 STEP 1693 H=1693:00 +CPU= 10.260
  22:26:41 STEP 1694 H=1694:00 +CPU= 11.010
  22:26:58 STEP 1695 H=1695:00 +CPU= 16.490
  22:27:09 STEP 1696 H=1696:00 +CPU= 11.039
  22:27:20 STEP 1697 H=1697:00 +CPU= 10.704
  22:27:37 STEP 1698 H=1698:00 +CPU= 16.863
  22:27:47 STEP 1699 H=1699:00 +CPU= 10.551
  22:27:58 STEP 1700 H=1700:00 +CPU= 10.965
  22:28:15 STEP 1701 H=1701:00 +CPU= 16.723
  22:28:26 STEP 1702 H=1702:00 +CPU= 11.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:53 STEP 1703 H=1703:00 +CPU= 11.856
  22:29:12 STEP 1704 H=1704:00 +CPU= 19.316
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_70.zip
Uploading the intermediate file: upload_file_70.zip
  22:29:23 STEP 1705 H=1705:00 +CPU= 10.464
  22:29:34 STEP 1706 H=1706:00 +CPU= 11.267
  22:29:50 STEP 1707 H=1707:00 +CPU= 16.451
  22:30:01 STEP 1708 H=1708:00 +CPU= 10.898
  22:30:12 STEP 1709 H=1709:00 +CPU= 10.694
  22:30:29 STEP 1710 H=1710:00 +CPU= 17.101
  22:30:40 STEP 1711 H=1711:00 +CPU= 10.870
  22:30:51 STEP 1712 H=1712:00 +CPU= 11.062
  22:31:08 STEP 1713 H=1713:00 +CPU= 16.737
  22:31:20 STEP 1714 H=1714:00 +CPU= 11.541
  22:31:30 STEP 1715 H=1715:00 +CPU= 10.638
  22:31:49 STEP 1716 H=1716:00 +CPU= 18.503
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001716
  22:31:59 STEP 1717 H=1717:00 +CPU= 10.552
  22:32:11 STEP 1718 H=1718:00 +CPU= 11.096
  22:32:29 STEP 1719 H=1719:00 +CPU= 18.768
  22:32:40 STEP 1720 H=1720:00 +CPU= 10.921
  22:32:51 STEP 1721 H=1721:00 +CPU= 10.810
  22:33:08 STEP 1722 H=1722:00 +CPU= 17.041
  22:33:19 STEP 1723 H=1723:00 +CPU= 10.692
  22:33:30 STEP 1724 H=1724:00 +CPU= 10.948
  22:33:47 STEP 1725 H=1725:00 +CPU= 16.925
  22:33:58 STEP 1726 H=1726:00 +CPU= 11.094
  22:34:11 STEP 1727 H=1727:00 +CPU= 11.719
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:34:45 STEP 1728 H=1728:00 +CPU= 20.632
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_71.zip
Uploading the intermediate file: upload_file_71.zip
  22:34:56 STEP 1729 H=1729:00 +CPU= 10.555
  22:35:09 STEP 1730 H=1730:00 +CPU= 11.375
  22:35:27 STEP 1731 H=1731:00 +CPU= 16.909
  22:35:38 STEP 1732 H=1732:00 +CPU= 10.885
  22:35:48 STEP 1733 H=1733:00 +CPU= 10.733
  22:36:05 STEP 1734 H=1734:00 +CPU= 16.867
  22:36:16 STEP 1735 H=1735:00 +CPU= 10.585
  22:36:27 STEP 1736 H=1736:00 +CPU= 10.846
  22:36:43 STEP 1737 H=1737:00 +CPU= 15.834
  22:36:53 STEP 1738 H=1738:00 +CPU= 10.442
  22:37:04 STEP 1739 H=1739:00 +CPU= 10.717
  22:37:23 STEP 1740 H=1740:00 +CPU= 18.719
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001740
  22:37:33 STEP 1741 H=1741:00 +CPU= 10.665
  22:37:45 STEP 1742 H=1742:00 +CPU= 11.173
  22:38:01 STEP 1743 H=1743:00 +CPU= 16.498
  22:38:12 STEP 1744 H=1744:00 +CPU= 11.058
  22:38:23 STEP 1745 H=1745:00 +CPU= 10.701
  22:38:40 STEP 1746 H=1746:00 +CPU= 17.187
  22:38:51 STEP 1747 H=1747:00 +CPU= 10.566
  22:39:02 STEP 1748 H=1748:00 +CPU= 10.879
  22:39:19 STEP 1749 H=1749:00 +CPU= 16.887
  22:39:30 STEP 1750 H=1750:00 +CPU= 10.907
  22:39:40 STEP 1751 H=1751:00 +CPU= 10.733
  22:40:01 STEP 1752 H=1752:00 +CPU= 20.380
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_72.zip
Uploading the intermediate file: upload_file_72.zip
  22:40:14 STEP 1753 H=1753:00 +CPU= 13.362
  22:40:26 STEP 1754 H=1754:00 +CPU= 11.183
  22:40:42 STEP 1755 H=1755:00 +CPU= 16.531
  22:40:53 STEP 1756 H=1756:00 +CPU= 11.077
  22:41:04 STEP 1757 H=1757:00 +CPU= 10.443
  22:41:21 STEP 1758 H=1758:00 +CPU= 16.937
  22:41:31 STEP 1759 H=1759:00 +CPU= 10.503
  22:41:43 STEP 1760 H=1760:00 +CPU= 11.193
  22:41:59 STEP 1761 H=1761:00 +CPU= 16.869
  22:42:10 STEP 1762 H=1762:00 +CPU= 10.845
  22:42:22 STEP 1763 H=1763:00 +CPU= 11.742
  22:42:46 STEP 1764 H=1764:00 +CPU= 24.141
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001764
  22:42:59 STEP 1765 H=1765:00 +CPU= 12.104
  22:43:10 STEP 1766 H=1766:00 +CPU= 11.093
  22:43:26 STEP 1767 H=1767:00 +CPU= 16.355
  22:43:37 STEP 1768 H=1768:00 +CPU= 10.568
  22:43:47 STEP 1769 H=1769:00 +CPU= 10.741
  22:44:04 STEP 1770 H=1770:00 +CPU= 16.836
  22:44:15 STEP 1771 H=1771:00 +CPU= 10.896
  22:44:26 STEP 1772 H=1772:00 +CPU= 11.033
  22:44:43 STEP 1773 H=1773:00 +CPU= 16.314
  22:44:57 STEP 1774 H=1774:00 +CPU= 13.938
  22:45:12 STEP 1775 H=1775:00 +CPU= 15.759
  22:45:32 STEP 1776 H=1776:00 +CPU= 19.556
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_73.zip
Uploading the intermediate file: upload_file_73.zip
  22:45:43 STEP 1777 H=1777:00 +CPU= 10.721
  22:45:54 STEP 1778 H=1778:00 +CPU= 10.898
  22:46:11 STEP 1779 H=1779:00 +CPU= 16.668
  22:46:22 STEP 1780 H=1780:00 +CPU= 10.993
  22:46:33 STEP 1781 H=1781:00 +CPU= 10.869
  22:46:50 STEP 1782 H=1782:00 +CPU= 17.055
  22:47:00 STEP 1783 H=1783:00 +CPU= 10.454
  22:47:13 STEP 1784 H=1784:00 +CPU= 10.985
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0258
(argv4) batchid: 967
(argv5) wuid: 12183902
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/0
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9f1538260466a9193194173f9ed56c34 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip
ic_ancil_file: ic_ancil_12183902
ifsdata_file: ifsdata_12183902
climate_data_file: clim_data_12183902
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc/slots/0/ic_ancil_12183902.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12183902.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4f0e1258061265fd68fcb5e19d1d2808 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip
Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12183902.xml
Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12183902.xml
-- Model is restarting --
Adjusting last_iter, 1785, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/0/progress_file_12183902.xml
last_cpu_time: 22180
upload_file_number: 74
last_iter: 1775
last_upload: 6393600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715
The child process has been launched with process id: 3409268
Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3409268:3409268] [20221231:224924:1672507164:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:52:03 STEP 1776 H=1776:00 +CPU= 21.190
  22:53:32 STEP 1777 H=1777:00 +CPU= 10.348
  22:53:43 STEP 1778 H=1778:00 +CPU= 10.243
  22:54:14 STEP 1779 H=1779:00 +CPU= 16.413
  22:55:40 STEP 1780 H=1780:00 +CPU= 11.727
  22:57:45 STEP 1781 H=1781:00 +CPU= 11.246
  22:59:51 STEP 1782 H=1782:00 +CPU= 17.954
  23:00:06 STEP 1783 H=1783:00 +CPU= 10.789
  23:01:21 STEP 1784 H=1784:00 +CPU= 10.952
  23:01:39 STEP 1785 H=1785:00 +CPU= 16.662
  23:06:51 STEP 1786 H=1786:00 +CPU= 12.006
  23:07:50 STEP 1787 H=1787:00 +CPU= 10.179
  23:08:18 STEP 1788 H=1788:00 +CPU= 17.916
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001788
  23:08:29 STEP 1789 H=1789:00 +CPU= 10.852
  23:08:40 STEP 1790 H=1790:00 +CPU= 11.119
  23:11:19 STEP 1791 H=1791:00 +CPU= 16.961
  23:12:28 STEP 1792 H=1792:00 +CPU= 10.893
  23:12:40 STEP 1793 H=1793:00 +CPU= 10.612
  23:12:57 STEP 1794 H=1794:00 +CPU= 17.107
  23:13:08 STEP 1795 H=1795:00 +CPU= 10.784
  23:13:20 STEP 1796 H=1796:00 +CPU= 11.361
  23:13:36 STEP 1797 H=1797:00 +CPU= 16.735
  23:13:48 STEP 1798 H=1798:00 +CPU= 11.223
  23:13:59 STEP 1799 H=1799:00 +CPU= 11.182
  23:14:26 STEP 1800 H=1800:00 +CPU= 20.093
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_74.zip
Uploading the intermediate file: upload_file_74.zip
  23:14:38 STEP 1801 H=1801:00 +CPU= 11.128
  23:14:49 STEP 1802 H=1802:00 +CPU= 11.072
  23:15:06 STEP 1803 H=1803:00 +CPU= 16.948
  23:15:17 STEP 1804 H=1804:00 +CPU= 11.201
  23:15:28 STEP 1805 H=1805:00 +CPU= 11.017
  23:15:46 STEP 1806 H=1806:00 +CPU= 17.252
  23:15:57 STEP 1807 H=1807:00 +CPU= 10.924
  23:16:08 STEP 1808 H=1808:00 +CPU= 11.209
  23:16:25 STEP 1809 H=1809:00 +CPU= 16.930
  23:16:36 STEP 1810 H=1810:00 +CPU= 11.403
  23:16:56 STEP 1811 H=1811:00 +CPU= 10.238
  23:17:19 STEP 1812 H=1812:00 +CPU= 16.990
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001812
  23:17:47 STEP 1813 H=1813:00 +CPU= 10.450
  23:17:59 STEP 1814 H=1814:00 +CPU= 11.148
  23:19:54 STEP 1815 H=1815:00 +CPU= 17.045
  23:20:05 STEP 1816 H=1816:00 +CPU= 11.361
  23:21:54 STEP 1817 H=1817:00 +CPU= 10.268
  23:22:13 STEP 1818 H=1818:00 +CPU= 16.669
  23:22:24 STEP 1819 H=1819:00 +CPU= 10.760
  23:23:09 STEP 1820 H=1820:00 +CPU= 10.352
  23:23:30 STEP 1821 H=1821:00 +CPU= 15.900
  23:23:41 STEP 1822 H=1822:00 +CPU= 10.308
  23:25:07 STEP 1823 H=1823:00 +CPU= 10.525
  23:27:12 STEP 1824 H=1824:00 +CPU= 19.546
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_75.zip
Uploading the intermediate file: upload_file_75.zip
  23:27:23 STEP 1825 H=1825:00 +CPU= 10.813
  23:27:42 STEP 1826 H=1826:00 +CPU= 11.354
  23:27:58 STEP 1827 H=1827:00 +CPU= 15.947
  23:28:09 STEP 1828 H=1828:00 +CPU= 10.576
  23:29:45 STEP 1829 H=1829:00 +CPU= 10.700
  23:31:14 STEP 1830 H=1830:00 +CPU= 17.083
  23:31:26 STEP 1831 H=1831:00 +CPU= 10.819
  23:31:37 STEP 1832 H=1832:00 +CPU= 10.986
  23:32:01 STEP 1833 H=1833:00 +CPU= 16.322
  23:32:12 STEP 1834 H=1834:00 +CPU= 11.277
  23:32:22 STEP 1835 H=1835:00 +CPU= 10.262
  23:32:44 STEP 1836 H=1836:00 +CPU= 18.238
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001836
  23:32:54 STEP 1837 H=1837:00 +CPU= 10.753
  23:33:26 STEP 1838 H=1838:00 +CPU= 10.347
  23:33:43 STEP 1839 H=1839:00 +CPU= 14.897
  23:33:53 STEP 1840 H=1840:00 +CPU=  9.856
  23:34:03 STEP 1841 H=1841:00 +CPU=  9.529
  23:34:18 STEP 1842 H=1842:00 +CPU= 15.328
  23:34:29 STEP 1843 H=1843:00 +CPU= 10.393
  23:34:39 STEP 1844 H=1844:00 +CPU= 10.735
  23:34:57 STEP 1845 H=1845:00 +CPU= 17.037
  23:35:13 STEP 1846 H=1846:00 +CPU= 10.355
  23:35:26 STEP 1847 H=1847:00 +CPU=  9.806
  23:36:34 STEP 1848 H=1848:00 +CPU= 17.902
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_76.zip
Uploading the intermediate file: upload_file_76.zip
  23:36:45 STEP 1849 H=1849:00 +CPU= 10.289
  23:37:26 STEP 1850 H=1850:00 +CPU= 10.783
  23:38:04 STEP 1851 H=1851:00 +CPU= 15.914
  23:38:14 STEP 1852 H=1852:00 +CPU=  9.784
  23:38:24 STEP 1853 H=1853:00 +CPU=  9.525
  23:38:39 STEP 1854 H=1854:00 +CPU= 15.335
  23:38:49 STEP 1855 H=1855:00 +CPU= 10.302
  23:39:00 STEP 1856 H=1856:00 +CPU= 10.757
  23:39:17 STEP 1857 H=1857:00 +CPU= 17.057
  23:39:43 STEP 1858 H=1858:00 +CPU= 10.431
  23:39:53 STEP 1859 H=1859:00 +CPU=  9.783
  23:40:25 STEP 1860 H=1860:00 +CPU= 17.231
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001860
  23:40:35 STEP 1861 H=1861:00 +CPU=  9.853
  23:40:45 STEP 1862 H=1862:00 +CPU=  9.856
  23:41:01 STEP 1863 H=1863:00 +CPU= 16.094
  23:41:12 STEP 1864 H=1864:00 +CPU= 10.783
  23:41:23 STEP 1865 H=1865:00 +CPU= 10.706
  23:44:07 STEP 1866 H=1866:00 +CPU= 18.551
  23:45:10 STEP 1867 H=1867:00 +CPU= 11.142
  23:45:29 STEP 1868 H=1868:00 +CPU= 10.884
  23:46:14 STEP 1869 H=1869:00 +CPU= 16.532
  23:46:34 STEP 1870 H=1870:00 +CPU= 10.948
  23:46:45 STEP 1871 H=1871:00 +CPU= 11.137
  23:47:26 STEP 1872 H=1872:00 +CPU= 19.931
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_77.zip
Uploading the intermediate file: upload_file_77.zip
  23:47:37 STEP 1873 H=1873:00 +CPU= 10.908
  23:47:48 STEP 1874 H=1874:00 +CPU= 11.166
  23:48:05 STEP 1875 H=1875:00 +CPU= 16.801
  23:48:16 STEP 1876 H=1876:00 +CPU= 11.166
  23:48:27 STEP 1877 H=1877:00 +CPU= 10.821
  23:48:44 STEP 1878 H=1878:00 +CPU= 17.114
  23:48:55 STEP 1879 H=1879:00 +CPU= 10.855
  23:49:06 STEP 1880 H=1880:00 +CPU= 11.122
  23:49:23 STEP 1881 H=1881:00 +CPU= 16.683
  23:49:34 STEP 1882 H=1882:00 +CPU= 11.140
  23:49:45 STEP 1883 H=1883:00 +CPU= 10.866
  23:50:05 STEP 1884 H=1884:00 +CPU= 18.435
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001884
  23:50:16 STEP 1885 H=1885:00 +CPU= 10.875
  23:50:27 STEP 1886 H=1886:00 +CPU= 11.162
  23:50:44 STEP 1887 H=1887:00 +CPU= 16.748
  23:50:55 STEP 1888 H=1888:00 +CPU= 11.106
  23:51:06 STEP 1889 H=1889:00 +CPU= 10.788
  23:51:23 STEP 1890 H=1890:00 +CPU= 16.336
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0258
(argv4) batchid: 967
(argv5) wuid: 12183902
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/0
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9f1538260466a9193194173f9ed56c34 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip
ic_ancil_file: ic_ancil_12183902
ifsdata_file: ifsdata_12183902
climate_data_file: clim_data_12183902
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc/slots/0/ic_ancil_12183902.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12183902.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4f0e1258061265fd68fcb5e19d1d2808 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip
Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12183902.xml
Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12183902.xml
-- Model is restarting --
Adjusting last_iter, 1891, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/0/progress_file_12183902.xml
last_cpu_time: 23670
upload_file_number: 78
last_iter: 1871
last_upload: 6739200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715
The child process has been launched with process id: 3424569
Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424569:3424569] [20221231:235421:1672511061:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:54:53 STEP 1872 H=1872:00 +CPU= 19.126
  23:55:43 STEP 1873 H=1873:00 +CPU= 10.080
  23:55:54 STEP 1874 H=1874:00 +CPU=  9.945
  23:56:14 STEP 1875 H=1875:00 +CPU= 14.941
  23:56:25 STEP 1876 H=1876:00 +CPU= 10.499
  23:56:36 STEP 1877 H=1877:00 +CPU= 10.355
  23:56:53 STEP 1878 H=1878:00 +CPU= 16.954
  23:57:23 STEP 1879 H=1879:00 +CPU= 10.235
  23:57:33 STEP 1880 H=1880:00 +CPU=  9.803
  23:57:48 STEP 1881 H=1881:00 +CPU= 14.928
  23:57:58 STEP 1882 H=1882:00 +CPU=  9.830
  23:58:08 STEP 1883 H=1883:00 +CPU=  9.507
  23:58:33 STEP 1884 H=1884:00 +CPU= 16.655
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001884
  23:58:43 STEP 1885 H=1885:00 +CPU= 10.283
  23:58:54 STEP 1886 H=1886:00 +CPU= 10.796
  00:01:21 STEP 1887 H=1887:00 +CPU= 17.994
  00:02:06 STEP 1888 H=1888:00 +CPU= 10.639
  00:02:18 STEP 1889 H=1889:00 +CPU=  9.785
  00:02:42 STEP 1890 H=1890:00 +CPU= 15.784
  00:02:52 STEP 1891 H=1891:00 +CPU=  9.386
  00:03:01 STEP 1892 H=1892:00 +CPU=  9.644
  00:03:16 STEP 1893 H=1893:00 +CPU= 14.745
  00:03:26 STEP 1894 H=1894:00 +CPU=  9.662
  00:03:37 STEP 1895 H=1895:00 +CPU=  9.756
  00:04:15 STEP 1896 H=1896:00 +CPU= 19.673
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_78.zip
Uploading the intermediate file: upload_file_78.zip
  00:05:22 STEP 1897 H=1897:00 +CPU= 10.781
  00:05:37 STEP 1898 H=1898:00 +CPU= 11.018
  00:05:54 STEP 1899 H=1899:00 +CPU= 17.151
  00:06:13 STEP 1900 H=1900:00 +CPU= 10.188
  00:06:23 STEP 1901 H=1901:00 +CPU=  9.503
  00:06:40 STEP 1902 H=1902:00 +CPU= 15.535
  00:06:51 STEP 1903 H=1903:00 +CPU= 10.596
  00:07:02 STEP 1904 H=1904:00 +CPU= 10.751
  00:07:17 STEP 1905 H=1905:00 +CPU= 15.623
  00:08:53 STEP 1906 H=1906:00 +CPU= 10.977
  00:09:04 STEP 1907 H=1907:00 +CPU= 10.524
  00:09:27 STEP 1908 H=1908:00 +CPU= 18.047
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001908
  00:09:38 STEP 1909 H=1909:00 +CPU= 10.241
  00:11:46 STEP 1910 H=1910:00 +CPU= 10.975
  00:12:06 STEP 1911 H=1911:00 +CPU= 16.910
  00:12:34 STEP 1912 H=1912:00 +CPU= 10.591
  00:12:45 STEP 1913 H=1913:00 +CPU=  9.917
  00:13:30 STEP 1914 H=1914:00 +CPU= 16.204
  00:13:40 STEP 1915 H=1915:00 +CPU=  9.879
  00:13:50 STEP 1916 H=1916:00 +CPU= 10.095
  00:14:10 STEP 1917 H=1917:00 +CPU= 15.845
  00:14:21 STEP 1918 H=1918:00 +CPU= 10.806
  00:14:33 STEP 1919 H=1919:00 +CPU= 11.136
  00:15:38 STEP 1920 H=1920:00 +CPU= 20.393
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_79.zip
Uploading the intermediate file: upload_file_79.zip
  00:15:49 STEP 1921 H=1921:00 +CPU= 11.055
  00:16:28 STEP 1922 H=1922:00 +CPU= 10.792
  00:16:51 STEP 1923 H=1923:00 +CPU= 16.080
  00:17:01 STEP 1924 H=1924:00 +CPU= 10.889
  00:19:03 STEP 1925 H=1925:00 +CPU= 11.423
  00:19:23 STEP 1926 H=1926:00 +CPU= 17.114
  00:19:34 STEP 1927 H=1927:00 +CPU= 10.671
  00:19:45 STEP 1928 H=1928:00 +CPU= 11.170
  00:20:45 STEP 1929 H=1929:00 +CPU= 16.826
  00:21:05 STEP 1930 H=1930:00 +CPU= 10.778
  00:21:16 STEP 1931 H=1931:00 +CPU= 10.535
  00:22:02 STEP 1932 H=1932:00 +CPU= 18.615
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001932
  00:22:12 STEP 1933 H=1933:00 +CPU= 10.499
  00:22:23 STEP 1934 H=1934:00 +CPU= 10.917
  00:23:28 STEP 1935 H=1935:00 +CPU= 16.816
  00:28:38 STEP 1936 H=1936:00 +CPU= 12.403
  00:30:16 STEP 1937 H=1937:00 +CPU= 10.419
  00:30:42 STEP 1938 H=1938:00 +CPU= 15.449
  00:30:52 STEP 1939 H=1939:00 +CPU=  9.531
  00:31:02 STEP 1940 H=1940:00 +CPU=  9.892
  00:31:17 STEP 1941 H=1941:00 +CPU= 15.089
  00:31:27 STEP 1942 H=1942:00 +CPU=  9.890
  00:31:38 STEP 1943 H=1943:00 +CPU= 10.071
  00:36:50 STEP 1944 H=1944:00 +CPU= 21.247
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_80.zip
Uploading the intermediate file: upload_file_80.zip
  00:38:14 STEP 1945 H=1945:00 +CPU= 11.148
  00:40:27 STEP 1946 H=1946:00 +CPU= 11.392
  00:41:56 STEP 1947 H=1947:00 +CPU= 17.135
  00:42:07 STEP 1948 H=1948:00 +CPU= 11.258
  00:42:18 STEP 1949 H=1949:00 +CPU= 11.098
  00:42:39 STEP 1950 H=1950:00 +CPU= 16.286
  00:42:49 STEP 1951 H=1951:00 +CPU=  9.759
  00:43:00 STEP 1952 H=1952:00 +CPU= 10.983
  00:44:22 STEP 1953 H=1953:00 +CPU= 16.902
  00:44:35 STEP 1954 H=1954:00 +CPU= 11.011
  00:46:35 STEP 1955 H=1955:00 +CPU= 10.920
  00:49:33 STEP 1956 H=1956:00 +CPU= 18.549
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001956
  00:49:45 STEP 1957 H=1957:00 +CPU= 10.838
  00:50:48 STEP 1958 H=1958:00 +CPU= 11.078
  00:51:11 STEP 1959 H=1959:00 +CPU= 15.092
  00:51:21 STEP 1960 H=1960:00 +CPU= 10.486
  00:51:31 STEP 1961 H=1961:00 +CPU= 10.341
  00:51:49 STEP 1962 H=1962:00 +CPU= 17.227
  00:52:13 STEP 1963 H=1963:00 +CPU= 10.683
  00:52:24 STEP 1964 H=1964:00 +CPU=  9.801
  00:52:40 STEP 1965 H=1965:00 +CPU= 15.032
  00:52:50 STEP 1966 H=1966:00 +CPU=  9.804
  00:53:03 STEP 1967 H=1967:00 +CPU=  9.840
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0258
(argv4) batchid: 967
(argv5) wuid: 12183902
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/0
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9f1538260466a9193194173f9ed56c34 to: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/0/oifs_43r3_ps_0258_1998050100_123_967_12183902.zip
ic_ancil_file: ic_ancil_12183902
ifsdata_file: ifsdata_12183902
climate_data_file: clim_data_12183902
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc/slots/0/ic_ancil_12183902.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/0/ic_ancil_12183902.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4f0e1258061265fd68fcb5e19d1d2808 to: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/0/ifsdata/ifsdata_12183902.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/0/159l_2/clim_data_12183902.zip
Checking for progress XML file: /var/lib/boinc/slots/0/progress_file_12183902.xml
Opened progress file ok : /var/lib/boinc/slots/0/progress_file_12183902.xml
-- Model is restarting --
Adjusting last_iter, 1968, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/0/progress_file_12183902.xml
last_cpu_time: 24898
upload_file_number: 81
last_iter: 1967
last_upload: 6998400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715
The child process has been launched with process id: 3424745
Executing the command: /var/lib/boinc/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:kishore-work:1:1:3424745:3424745] [20230101:005724:1672514844:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:58:33 STEP 1968 H=1968:00 +CPU= 19.168
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_81.zip
Uploading the intermediate file: upload_file_81.zip
  01:00:48 STEP 1969 H=1969:00 +CPU= 11.036
  01:01:39 STEP 1970 H=1970:00 +CPU= 11.464
  01:01:56 STEP 1971 H=1971:00 +CPU= 16.517
  01:03:11 STEP 1972 H=1972:00 +CPU= 11.553
  01:03:22 STEP 1973 H=1973:00 +CPU= 10.963
  01:03:50 STEP 1974 H=1974:00 +CPU= 16.764
  01:05:21 STEP 1975 H=1975:00 +CPU= 10.418
  01:05:57 STEP 1976 H=1976:00 +CPU= 10.805
  01:08:29 STEP 1977 H=1977:00 +CPU= 22.318
  01:08:41 STEP 1978 H=1978:00 +CPU= 11.118
  01:08:52 STEP 1979 H=1979:00 +CPU= 11.121
  01:11:30 STEP 1980 H=1980:00 +CPU= 18.544
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001980
  01:12:46 STEP 1981 H=1981:00 +CPU= 10.792
  01:12:59 STEP 1982 H=1982:00 +CPU= 11.244
  01:13:36 STEP 1983 H=1983:00 +CPU= 16.801
  01:14:11 STEP 1984 H=1984:00 +CPU= 11.103
  01:14:21 STEP 1985 H=1985:00 +CPU= 10.503
  01:14:39 STEP 1986 H=1986:00 +CPU= 17.452
  01:15:08 STEP 1987 H=1987:00 +CPU= 10.747
  01:15:19 STEP 1988 H=1988:00 +CPU= 11.138
  01:15:36 STEP 1989 H=1989:00 +CPU= 17.073
  01:15:47 STEP 1990 H=1990:00 +CPU= 10.999
  01:15:58 STEP 1991 H=1991:00 +CPU= 11.204
  01:16:28 STEP 1992 H=1992:00 +CPU= 19.941
Moving to projects directory: /var/lib/boinc/slots/0/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/0/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/0/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12183902/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0258_1998050100_123_967_12183902_0_r1003190715_82.zip
Uploading the intermediate file: upload_file_82.zip
  01:16:39 STEP 1993 H=1993:00 +CPU= 10.865
  01:16:50 STEP 1994 H=1994:00 +CPU= 11.162
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:04:14 1533721 22269101 oifs_43r3_ps_0258_1998050100_123_967_12183902_0 5,616,000 19,263 0.0034
09 May 2023 16:02:42 1533721 22269101 oifs_43r3_ps_0258_1998050100_123_967_12183902_0 3,974,400 13,758 0.0035
09 May 2023 16:01:52 1533721 22269101 oifs_43r3_ps_0258_1998050100_123_967_12183902_0 2,678,400 9,307 0.0035
09 May 2023 16:01:48 1533721 22269101 oifs_43r3_ps_0258_1998050100_123_967_12183902_0 1,728,000 5,900 0.0034
09 May 2023 16:01:44 1533721 22269101 oifs_43r3_ps_0258_1998050100_123_967_12183902_0 864,000 2,879 0.0033


©2024 climateprediction.net