climateprediction.net home page
Task 22276783

Task 22276783

Name oifs_43r3_ps_0476_2005050100_123_974_12191120_0
Workunit 12191120
Created 23 Dec 2022, 10:40:20 UTC
Sent 30 Dec 2022, 19:54:56 UTC
Report deadline 29 Jan 2023, 19:54:56 UTC
Received 2 Jan 2023, 20:16:18 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1506016
Run time 19 hours 35 min 45 sec
CPU time 17 hours 47 min 53 sec
Validate state Invalid
Credit 9,797.76
Device peak FLOPS 5.45 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.88 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,352.82 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:3034:3034] [20230101:122550:1672604750:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:26:37 STEP 1704 H=1704:00 +CPU= 39.873
  12:26:59 STEP 1705 H=1705:00 +CPU= 22.661
  12:27:23 STEP 1706 H=1706:00 +CPU= 23.358
  12:27:57 STEP 1707 H=1707:00 +CPU= 34.637
  12:28:21 STEP 1708 H=1708:00 +CPU= 23.038
  12:28:44 STEP 1709 H=1709:00 +CPU= 22.388
  12:29:47 STEP 1710 H=1710:00 +CPU= 37.279
  12:30:44 STEP 1711 H=1711:00 +CPU= 25.971
  12:31:49 STEP 1712 H=1712:00 +CPU= 27.238
  12:33:15 STEP 1713 H=1713:00 +CPU= 41.442
  12:34:14 STEP 1714 H=1714:00 +CPU= 26.345
  12:34:51 STEP 1715 H=1715:00 +CPU= 23.846
  12:35:40 STEP 1716 H=1716:00 +CPU= 39.614
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001716
  12:36:08 STEP 1717 H=1717:00 +CPU= 23.495
  12:36:36 STEP 1718 H=1718:00 +CPU= 23.973
  12:37:22 STEP 1719 H=1719:00 +CPU= 36.305
  12:37:53 STEP 1720 H=1720:00 +CPU= 24.448
  12:38:23 STEP 1721 H=1721:00 +CPU= 23.411
  12:39:08 STEP 1722 H=1722:00 +CPU= 36.699
  12:39:37 STEP 1723 H=1723:00 +CPU= 23.462
  12:40:06 STEP 1724 H=1724:00 +CPU= 23.900
  12:40:46 STEP 1725 H=1725:00 +CPU= 35.520
  12:41:13 STEP 1726 H=1726:00 +CPU= 23.768
  12:41:40 STEP 1727 H=1727:00 +CPU= 23.968
  12:42:29 STEP 1728 H=1728:00 +CPU= 42.438
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_71.zip
Uploading the intermediate file: upload_file_71.zip
  12:42:55 STEP 1729 H=1729:00 +CPU= 23.042
  12:43:21 STEP 1730 H=1730:00 +CPU= 23.681
  12:44:01 STEP 1731 H=1731:00 +CPU= 35.483
  12:44:27 STEP 1732 H=1732:00 +CPU= 23.543
  12:44:53 STEP 1733 H=1733:00 +CPU= 23.096
  12:45:33 STEP 1734 H=1734:00 +CPU= 36.162
  12:45:59 STEP 1735 H=1735:00 +CPU= 22.886
  12:46:25 STEP 1736 H=1736:00 +CPU= 23.635
  12:47:05 STEP 1737 H=1737:00 +CPU= 35.407
  12:47:32 STEP 1738 H=1738:00 +CPU= 23.949
  12:47:57 STEP 1739 H=1739:00 +CPU= 22.916
  12:48:41 STEP 1740 H=1740:00 +CPU= 39.407
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001740
  12:49:06 STEP 1741 H=1741:00 +CPU= 22.894
  12:49:33 STEP 1742 H=1742:00 +CPU= 23.749
  12:50:12 STEP 1743 H=1743:00 +CPU= 35.504
  12:50:39 STEP 1744 H=1744:00 +CPU= 23.900
  12:51:04 STEP 1745 H=1745:00 +CPU= 23.226
  12:51:44 STEP 1746 H=1746:00 +CPU= 36.234
  12:52:10 STEP 1747 H=1747:00 +CPU= 23.084
  12:52:37 STEP 1748 H=1748:00 +CPU= 23.763
  12:53:16 STEP 1749 H=1749:00 +CPU= 35.272
  12:53:42 STEP 1750 H=1750:00 +CPU= 23.499
  12:54:08 STEP 1751 H=1751:00 +CPU= 23.612
  12:54:56 STEP 1752 H=1752:00 +CPU= 41.964
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_72.zip
Uploading the intermediate file: upload_file_72.zip
  12:55:22 STEP 1753 H=1753:00 +CPU= 22.922
  12:55:49 STEP 1754 H=1754:00 +CPU= 23.695
  12:56:28 STEP 1755 H=1755:00 +CPU= 35.060
  12:56:54 STEP 1756 H=1756:00 +CPU= 23.567
  12:57:19 STEP 1757 H=1757:00 +CPU= 23.061
  12:58:00 STEP 1758 H=1758:00 +CPU= 35.866
  12:58:26 STEP 1759 H=1759:00 +CPU= 22.910
  12:58:52 STEP 1760 H=1760:00 +CPU= 23.440
  12:59:32 STEP 1761 H=1761:00 +CPU= 35.398
  12:59:58 STEP 1762 H=1762:00 +CPU= 23.625
  13:00:24 STEP 1763 H=1763:00 +CPU= 23.062
  13:01:11 STEP 1764 H=1764:00 +CPU= 39.820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001764
  13:01:36 STEP 1765 H=1765:00 +CPU= 22.954
  13:02:03 STEP 1766 H=1766:00 +CPU= 23.702
  13:02:42 STEP 1767 H=1767:00 +CPU= 35.262
  13:03:08 STEP 1768 H=1768:00 +CPU= 23.547
  13:03:33 STEP 1769 H=1769:00 +CPU= 22.697
  13:04:14 STEP 1770 H=1770:00 +CPU= 36.076
  13:04:39 STEP 1771 H=1771:00 +CPU= 23.038
  13:05:06 STEP 1772 H=1772:00 +CPU= 23.899
  13:05:46 STEP 1773 H=1773:00 +CPU= 35.290
  13:06:12 STEP 1774 H=1774:00 +CPU= 23.770
  13:06:39 STEP 1775 H=1775:00 +CPU= 23.816
  13:07:26 STEP 1776 H=1776:00 +CPU= 42.068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_73.zip
Uploading the intermediate file: upload_file_73.zip
  13:07:52 STEP 1777 H=1777:00 +CPU= 23.219
  13:08:21 STEP 1778 H=1778:00 +CPU= 23.972
  13:08:59 STEP 1779 H=1779:00 +CPU= 35.373
  13:09:26 STEP 1780 H=1780:00 +CPU= 23.859
  13:09:51 STEP 1781 H=1781:00 +CPU= 22.943
  13:10:31 STEP 1782 H=1782:00 +CPU= 36.176
  13:10:57 STEP 1783 H=1783:00 +CPU= 22.999
  13:11:23 STEP 1784 H=1784:00 +CPU= 23.943
  13:12:02 STEP 1785 H=1785:00 +CPU= 35.329
  13:12:29 STEP 1786 H=1786:00 +CPU= 23.920
  13:12:59 STEP 1787 H=1787:00 +CPU= 23.637
  13:13:45 STEP 1788 H=1788:00 +CPU= 39.793
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001788
  13:14:11 STEP 1789 H=1789:00 +CPU= 23.092
  13:14:38 STEP 1790 H=1790:00 +CPU= 23.989
  13:15:17 STEP 1791 H=1791:00 +CPU= 35.180
  13:15:43 STEP 1792 H=1792:00 +CPU= 23.832
  13:16:10 STEP 1793 H=1793:00 +CPU= 23.366
  13:16:54 STEP 1794 H=1794:00 +CPU= 36.814
  13:17:19 STEP 1795 H=1795:00 +CPU= 23.165
  13:17:48 STEP 1796 H=1796:00 +CPU= 24.226
  13:18:33 STEP 1797 H=1797:00 +CPU= 36.048
  13:18:59 STEP 1798 H=1798:00 +CPU= 23.668
  13:19:26 STEP 1799 H=1799:00 +CPU= 24.118
  13:20:13 STEP 1800 H=1800:00 +CPU= 41.872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_74.zip
Uploading the intermediate file: upload_file_74.zip
  13:20:39 STEP 1801 H=1801:00 +CPU= 23.092
  13:21:06 STEP 1802 H=1802:00 +CPU= 23.831
  13:21:45 STEP 1803 H=1803:00 +CPU= 35.310
  13:22:12 STEP 1804 H=1804:00 +CPU= 23.885
  13:22:37 STEP 1805 H=1805:00 +CPU= 22.965
  13:23:18 STEP 1806 H=1806:00 +CPU= 36.337
  13:23:43 STEP 1807 H=1807:00 +CPU= 22.876
  13:24:09 STEP 1808 H=1808:00 +CPU= 23.633
  13:24:48 STEP 1809 H=1809:00 +CPU= 35.197
  13:25:14 STEP 1810 H=1810:00 +CPU= 23.743
  13:25:40 STEP 1811 H=1811:00 +CPU= 23.160
  13:26:24 STEP 1812 H=1812:00 +CPU= 39.380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001812
  13:26:49 STEP 1813 H=1813:00 +CPU= 22.817
  13:27:15 STEP 1814 H=1814:00 +CPU= 23.787
  13:27:54 STEP 1815 H=1815:00 +CPU= 35.202
  13:28:21 STEP 1816 H=1816:00 +CPU= 23.886
  13:28:46 STEP 1817 H=1817:00 +CPU= 23.234
  13:29:26 STEP 1818 H=1818:00 +CPU= 36.147
  13:29:51 STEP 1819 H=1819:00 +CPU= 22.789
  13:30:18 STEP 1820 H=1820:00 +CPU= 23.505
  13:30:59 STEP 1821 H=1821:00 +CPU= 35.368
  13:31:24 STEP 1822 H=1822:00 +CPU= 23.401
  13:31:51 STEP 1823 H=1823:00 +CPU= 23.854
  13:32:38 STEP 1824 H=1824:00 +CPU= 42.038
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_75.zip
Uploading the intermediate file: upload_file_75.zip
  13:33:07 STEP 1825 H=1825:00 +CPU= 23.214
  13:33:33 STEP 1826 H=1826:00 +CPU= 23.758
  13:34:13 STEP 1827 H=1827:00 +CPU= 35.172
  13:34:40 STEP 1828 H=1828:00 +CPU= 23.755
  13:35:05 STEP 1829 H=1829:00 +CPU= 22.983
  13:35:45 STEP 1830 H=1830:00 +CPU= 35.936
  13:36:11 STEP 1831 H=1831:00 +CPU= 23.087
  13:36:38 STEP 1832 H=1832:00 +CPU= 23.738
  13:37:17 STEP 1833 H=1833:00 +CPU= 34.974
  13:37:44 STEP 1834 H=1834:00 +CPU= 23.750
  13:38:11 STEP 1835 H=1835:00 +CPU= 23.089
  13:38:55 STEP 1836 H=1836:00 +CPU= 39.220
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001836
  13:39:22 STEP 1837 H=1837:00 +CPU= 23.086
  13:39:48 STEP 1838 H=1838:00 +CPU= 23.707
  13:40:27 STEP 1839 H=1839:00 +CPU= 35.285
  13:40:54 STEP 1840 H=1840:00 +CPU= 23.589
  13:41:20 STEP 1841 H=1841:00 +CPU= 22.852
  13:41:59 STEP 1842 H=1842:00 +CPU= 36.006
  13:42:25 STEP 1843 H=1843:00 +CPU= 22.901
  13:42:52 STEP 1844 H=1844:00 +CPU= 23.686
  13:43:32 STEP 1845 H=1845:00 +CPU= 35.330
  13:43:58 STEP 1846 H=1846:00 +CPU= 23.438
  13:44:24 STEP 1847 H=1847:00 +CPU= 23.708
  13:45:10 STEP 1848 H=1848:00 +CPU= 41.508
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_76.zip
Uploading the intermediate file: upload_file_76.zip
  13:45:36 STEP 1849 H=1849:00 +CPU= 22.904
  13:46:02 STEP 1850 H=1850:00 +CPU= 23.573
  13:46:41 STEP 1851 H=1851:00 +CPU= 35.111
  13:47:07 STEP 1852 H=1852:00 +CPU= 23.580
  13:47:32 STEP 1853 H=1853:00 +CPU= 22.849
  13:48:12 STEP 1854 H=1854:00 +CPU= 36.111
  13:48:37 STEP 1855 H=1855:00 +CPU= 22.894
  13:49:03 STEP 1856 H=1856:00 +CPU= 23.596
  13:49:42 STEP 1857 H=1857:00 +CPU= 35.226
  13:50:08 STEP 1858 H=1858:00 +CPU= 23.503
  13:50:33 STEP 1859 H=1859:00 +CPU= 22.860
  13:51:17 STEP 1860 H=1860:00 +CPU= 39.335
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001860
  13:51:42 STEP 1861 H=1861:00 +CPU= 22.988
  13:52:08 STEP 1862 H=1862:00 +CPU= 23.390
  13:52:48 STEP 1863 H=1863:00 +CPU= 35.298
  13:53:15 STEP 1864 H=1864:00 +CPU= 23.602
  13:53:39 STEP 1865 H=1865:00 +CPU= 22.683
  13:54:19 STEP 1866 H=1866:00 +CPU= 35.826
  13:54:45 STEP 1867 H=1867:00 +CPU= 22.884
  13:55:11 STEP 1868 H=1868:00 +CPU= 23.448
  13:55:50 STEP 1869 H=1869:00 +CPU= 35.142
  13:56:17 STEP 1870 H=1870:00 +CPU= 23.572
  13:56:43 STEP 1871 H=1871:00 +CPU= 23.736
  13:57:30 STEP 1872 H=1872:00 +CPU= 41.744
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_77.zip
Uploading the intermediate file: upload_file_77.zip
  13:57:56 STEP 1873 H=1873:00 +CPU= 22.770
  13:58:22 STEP 1874 H=1874:00 +CPU= 23.569
  13:59:02 STEP 1875 H=1875:00 +CPU= 35.292
  13:59:28 STEP 1876 H=1876:00 +CPU= 23.645
  13:59:53 STEP 1877 H=1877:00 +CPU= 22.794
  14:00:34 STEP 1878 H=1878:00 +CPU= 35.947
  14:00:59 STEP 1879 H=1879:00 +CPU= 22.872
  14:01:26 STEP 1880 H=1880:00 +CPU= 23.607
  14:02:06 STEP 1881 H=1881:00 +CPU= 35.351
  14:02:32 STEP 1882 H=1882:00 +CPU= 23.650
  14:02:58 STEP 1883 H=1883:00 +CPU= 22.829
  14:03:41 STEP 1884 H=1884:00 +CPU= 39.034
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001884
  14:04:07 STEP 1885 H=1885:00 +CPU= 23.052
  14:04:34 STEP 1886 H=1886:00 +CPU= 23.697
  14:05:13 STEP 1887 H=1887:00 +CPU= 35.458
  14:05:39 STEP 1888 H=1888:00 +CPU= 23.572
  14:06:05 STEP 1889 H=1889:00 +CPU= 22.781
  14:06:45 STEP 1890 H=1890:00 +CPU= 35.925
  14:07:10 STEP 1891 H=1891:00 +CPU= 22.924
  14:07:36 STEP 1892 H=1892:00 +CPU= 23.446
  14:08:15 STEP 1893 H=1893:00 +CPU= 35.512
  14:08:41 STEP 1894 H=1894:00 +CPU= 23.419
  14:09:07 STEP 1895 H=1895:00 +CPU= 23.758
  14:09:54 STEP 1896 H=1896:00 +CPU= 41.674
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_78.zip
Uploading the intermediate file: upload_file_78.zip
  14:10:20 STEP 1897 H=1897:00 +CPU= 22.690
  14:10:46 STEP 1898 H=1898:00 +CPU= 23.545
  14:11:25 STEP 1899 H=1899:00 +CPU= 35.151
  14:11:51 STEP 1900 H=1900:00 +CPU= 23.631
  14:12:16 STEP 1901 H=1901:00 +CPU= 22.989
  14:12:56 STEP 1902 H=1902:00 +CPU= 36.022
  14:13:22 STEP 1903 H=1903:00 +CPU= 22.885
  14:13:49 STEP 1904 H=1904:00 +CPU= 23.825
  14:14:27 STEP 1905 H=1905:00 +CPU= 34.584
  14:14:53 STEP 1906 H=1906:00 +CPU= 23.728
  14:15:19 STEP 1907 H=1907:00 +CPU= 23.179
  14:16:03 STEP 1908 H=1908:00 +CPU= 39.394
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001908
  14:16:28 STEP 1909 H=1909:00 +CPU= 23.106
  14:16:54 STEP 1910 H=1910:00 +CPU= 23.668
  14:17:34 STEP 1911 H=1911:00 +CPU= 35.374
  14:18:01 STEP 1912 H=1912:00 +CPU= 23.708
  14:18:26 STEP 1913 H=1913:00 +CPU= 22.867
  14:19:06 STEP 1914 H=1914:00 +CPU= 36.105
  14:19:31 STEP 1915 H=1915:00 +CPU= 22.896
  14:19:58 STEP 1916 H=1916:00 +CPU= 23.519
  14:20:37 STEP 1917 H=1917:00 +CPU= 35.329
  14:21:04 STEP 1918 H=1918:00 +CPU= 23.694
  14:21:30 STEP 1919 H=1919:00 +CPU= 23.681
  14:22:17 STEP 1920 H=1920:00 +CPU= 41.745
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_79.zip
Uploading the intermediate file: upload_file_79.zip
  14:22:43 STEP 1921 H=1921:00 +CPU= 22.818
  14:23:10 STEP 1922 H=1922:00 +CPU= 23.731
  14:23:49 STEP 1923 H=1923:00 +CPU= 35.405
  14:24:16 STEP 1924 H=1924:00 +CPU= 23.908
  14:24:41 STEP 1925 H=1925:00 +CPU= 22.840
  14:25:21 STEP 1926 H=1926:00 +CPU= 36.399
  14:25:47 STEP 1927 H=1927:00 +CPU= 22.949
  14:26:13 STEP 1928 H=1928:00 +CPU= 23.704
  14:26:52 STEP 1929 H=1929:00 +CPU= 35.203
  14:27:18 STEP 1930 H=1930:00 +CPU= 23.638
  14:27:44 STEP 1931 H=1931:00 +CPU= 22.951
  14:28:27 STEP 1932 H=1932:00 +CPU= 39.416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001932
  14:28:53 STEP 1933 H=1933:00 +CPU= 23.068
  14:29:19 STEP 1934 H=1934:00 +CPU= 23.724
  14:29:55 STEP 1935 H=1935:00 +CPU= 32.392
  14:30:22 STEP 1936 H=1936:00 +CPU= 23.326
  14:30:47 STEP 1937 H=1937:00 +CPU= 22.662
  14:31:27 STEP 1938 H=1938:00 +CPU= 35.152
  14:31:54 STEP 1939 H=1939:00 +CPU= 22.672
  14:32:20 STEP 1940 H=1940:00 +CPU= 23.332
  14:33:00 STEP 1941 H=1941:00 +CPU= 34.903
  14:33:28 STEP 1942 H=1942:00 +CPU= 23.550
  14:33:54 STEP 1943 H=1943:00 +CPU= 23.421
  14:34:43 STEP 1944 H=1944:00 +CPU= 41.436
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_80.zip
Uploading the intermediate file: upload_file_80.zip
Uploading trickle at timestep: 6998400
  14:35:09 STEP 1945 H=1945:00 +CPU= 22.869
  14:35:35 STEP 1946 H=1946:00 +CPU= 23.442
  14:36:15 STEP 1947 H=1947:00 +CPU= 34.961
  14:36:41 STEP 1948 H=1948:00 +CPU= 23.330
  14:37:07 STEP 1949 H=1949:00 +CPU= 22.735
  14:37:47 STEP 1950 H=1950:00 +CPU= 35.501
  14:38:12 STEP 1951 H=1951:00 +CPU= 22.635
  14:38:39 STEP 1952 H=1952:00 +CPU= 23.369
  14:39:17 STEP 1953 H=1953:00 +CPU= 34.802
  14:39:43 STEP 1954 H=1954:00 +CPU= 23.294
  14:40:08 STEP 1955 H=1955:00 +CPU= 22.595
  14:40:51 STEP 1956 H=1956:00 +CPU= 38.814
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001956
  14:41:17 STEP 1957 H=1957:00 +CPU= 22.658
  14:41:43 STEP 1958 H=1958:00 +CPU= 23.275
  14:42:21 STEP 1959 H=1959:00 +CPU= 34.475
  14:42:47 STEP 1960 H=1960:00 +CPU= 23.461
  14:43:13 STEP 1961 H=1961:00 +CPU= 22.739
  14:43:53 STEP 1962 H=1962:00 +CPU= 35.468
  14:44:18 STEP 1963 H=1963:00 +CPU= 22.466
  14:44:43 STEP 1964 H=1964:00 +CPU= 23.212
  14:45:22 STEP 1965 H=1965:00 +CPU= 34.744
  14:45:47 STEP 1966 H=1966:00 +CPU= 23.050
  14:46:13 STEP 1967 H=1967:00 +CPU= 23.225
  14:46:59 STEP 1968 H=1968:00 +CPU= 41.332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_81.zip
Uploading the intermediate file: upload_file_81.zip
  14:47:24 STEP 1969 H=1969:00 +CPU= 22.623
  14:47:50 STEP 1970 H=1970:00 +CPU= 23.044
  14:48:29 STEP 1971 H=1971:00 +CPU= 34.742
  14:48:54 STEP 1972 H=1972:00 +CPU= 23.154
  14:49:19 STEP 1973 H=1973:00 +CPU= 22.476
  14:50:01 STEP 1974 H=1974:00 +CPU= 35.792
  14:50:26 STEP 1975 H=1975:00 +CPU= 22.696
  14:50:52 STEP 1976 H=1976:00 +CPU= 23.352
  14:51:30 STEP 1977 H=1977:00 +CPU= 34.722
  14:51:56 STEP 1978 H=1978:00 +CPU= 23.299
  14:52:21 STEP 1979 H=1979:00 +CPU= 22.529
  14:53:05 STEP 1980 H=1980:00 +CPU= 38.885
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001980
  14:53:29 STEP 1981 H=1981:00 +CPU= 22.668
  14:53:55 STEP 1982 H=1982:00 +CPU= 23.447
  14:54:34 STEP 1983 H=1983:00 +CPU= 35.011
  14:55:00 STEP 1984 H=1984:00 +CPU= 23.286
  14:55:24 STEP 1985 H=1985:00 +CPU= 22.588
  14:56:05 STEP 1986 H=1986:00 +CPU= 35.652
  14:56:31 STEP 1987 H=1987:00 +CPU= 22.527
  14:56:56 STEP 1988 H=1988:00 +CPU= 23.304
  14:57:35 STEP 1989 H=1989:00 +CPU= 34.938
  14:58:00 STEP 1990 H=1990:00 +CPU= 23.207
  14:58:26 STEP 1991 H=1991:00 +CPU= 23.508
  14:59:13 STEP 1992 H=1992:00 +CPU= 41.458
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_82.zip
Uploading the intermediate file: upload_file_82.zip
  14:59:39 STEP 1993 H=1993:00 +CPU= 22.566
  15:00:05 STEP 1994 H=1994:00 +CPU= 23.189
  15:00:45 STEP 1995 H=1995:00 +CPU= 34.949
  15:01:11 STEP 1996 H=1996:00 +CPU= 23.373
  15:01:36 STEP 1997 H=1997:00 +CPU= 22.466
  15:02:16 STEP 1998 H=1998:00 +CPU= 35.610
  15:02:41 STEP 1999 H=1999:00 +CPU= 22.566
  15:03:07 STEP 2000 H=2000:00 +CPU= 23.381
  15:03:46 STEP 2001 H=2001:00 +CPU= 34.896
  15:04:12 STEP 2002 H=2002:00 +CPU= 23.387
  15:04:38 STEP 2003 H=2003:00 +CPU= 22.688
  15:05:24 STEP 2004 H=2004:00 +CPU= 38.995
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002004
  15:05:49 STEP 2005 H=2005:00 +CPU= 22.554
  15:06:15 STEP 2006 H=2006:00 +CPU= 23.452
  15:06:53 STEP 2007 H=2007:00 +CPU= 34.907
  15:07:19 STEP 2008 H=2008:00 +CPU= 23.279
  15:07:44 STEP 2009 H=2009:00 +CPU= 22.517
  15:08:24 STEP 2010 H=2010:00 +CPU= 35.802
  15:08:49 STEP 2011 H=2011:00 +CPU= 22.598
  15:09:15 STEP 2012 H=2012:00 +CPU= 23.459
  15:09:53 STEP 2013 H=2013:00 +CPU= 34.658
  15:10:19 STEP 2014 H=2014:00 +CPU= 23.214
  15:10:45 STEP 2015 H=2015:00 +CPU= 23.420
  15:11:31 STEP 2016 H=2016:00 +CPU= 41.278
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_83.zip
Uploading the intermediate file: upload_file_83.zip
  15:11:56 STEP 2017 H=2017:00 +CPU= 22.604
  15:12:22 STEP 2018 H=2018:00 +CPU= 23.305
  15:13:00 STEP 2019 H=2019:00 +CPU= 34.609
  15:13:26 STEP 2020 H=2020:00 +CPU= 23.312
  15:13:51 STEP 2021 H=2021:00 +CPU= 22.441
  15:14:30 STEP 2022 H=2022:00 +CPU= 35.701
  15:14:55 STEP 2023 H=2023:00 +CPU= 22.496
  15:15:21 STEP 2024 H=2024:00 +CPU= 23.259
  15:15:59 STEP 2025 H=2025:00 +CPU= 34.771
  15:16:25 STEP 2026 H=2026:00 +CPU= 23.281
  15:16:50 STEP 2027 H=2027:00 +CPU= 22.575
  15:17:33 STEP 2028 H=2028:00 +CPU= 38.979
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002028
  15:17:58 STEP 2029 H=2029:00 +CPU= 22.647
  15:18:24 STEP 2030 H=2030:00 +CPU= 23.363
  15:19:02 STEP 2031 H=2031:00 +CPU= 34.819
  15:19:28 STEP 2032 H=2032:00 +CPU= 23.481
  15:19:53 STEP 2033 H=2033:00 +CPU= 22.539
  15:20:34 STEP 2034 H=2034:00 +CPU= 36.082
  15:20:59 STEP 2035 H=2035:00 +CPU= 22.550
  15:21:25 STEP 2036 H=2036:00 +CPU= 23.311
  15:22:04 STEP 2037 H=2037:00 +CPU= 34.728
  15:22:29 STEP 2038 H=2038:00 +CPU= 23.225
  15:22:55 STEP 2039 H=2039:00 +CPU= 23.328
  15:23:41 STEP 2040 H=2040:00 +CPU= 41.410
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_84.zip
Uploading the intermediate file: upload_file_84.zip
  15:24:07 STEP 2041 H=2041:00 +CPU= 22.568
  15:24:32 STEP 2042 H=2042:00 +CPU= 23.146
  15:25:13 STEP 2043 H=2043:00 +CPU= 35.119
  15:25:38 STEP 2044 H=2044:00 +CPU= 23.205
  15:26:03 STEP 2045 H=2045:00 +CPU= 22.735
  15:26:43 STEP 2046 H=2046:00 +CPU= 35.742
  15:27:09 STEP 2047 H=2047:00 +CPU= 22.581
  15:27:35 STEP 2048 H=2048:00 +CPU= 23.404
  15:28:13 STEP 2049 H=2049:00 +CPU= 34.707
  15:28:39 STEP 2050 H=2050:00 +CPU= 23.333
  15:29:03 STEP 2051 H=2051:00 +CPU= 22.628
  15:29:46 STEP 2052 H=2052:00 +CPU= 38.555
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002052
  15:30:12 STEP 2053 H=2053:00 +CPU= 22.683
  15:30:40 STEP 2054 H=2054:00 +CPU= 23.467
  15:31:19 STEP 2055 H=2055:00 +CPU= 35.044
  15:31:47 STEP 2056 H=2056:00 +CPU= 23.442
  15:32:13 STEP 2057 H=2057:00 +CPU= 22.632
  15:32:53 STEP 2058 H=2058:00 +CPU= 35.533
  15:33:22 STEP 2059 H=2059:00 +CPU= 22.952
  15:33:47 STEP 2060 H=2060:00 +CPU= 23.369
  15:34:27 STEP 2061 H=2061:00 +CPU= 35.015
  15:34:54 STEP 2062 H=2062:00 +CPU= 23.583
  15:35:20 STEP 2063 H=2063:00 +CPU= 23.446
  15:36:08 STEP 2064 H=2064:00 +CPU= 41.392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_85.zip
Uploading the intermediate file: upload_file_85.zip
  15:36:34 STEP 2065 H=2065:00 +CPU= 22.655
  15:37:00 STEP 2066 H=2066:00 +CPU= 23.224
  15:37:38 STEP 2067 H=2067:00 +CPU= 34.860
  15:38:05 STEP 2068 H=2068:00 +CPU= 23.279
  15:38:30 STEP 2069 H=2069:00 +CPU= 22.543
  15:39:09 STEP 2070 H=2070:00 +CPU= 35.729
  15:39:34 STEP 2071 H=2071:00 +CPU= 22.719
  15:40:00 STEP 2072 H=2072:00 +CPU= 23.337
  15:40:39 STEP 2073 H=2073:00 +CPU= 34.739
  15:41:05 STEP 2074 H=2074:00 +CPU= 23.379
  15:41:30 STEP 2075 H=2075:00 +CPU= 22.910
  15:42:13 STEP 2076 H=2076:00 +CPU= 38.750
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002076
  15:42:38 STEP 2077 H=2077:00 +CPU= 22.501
  15:43:04 STEP 2078 H=2078:00 +CPU= 23.264
  15:43:44 STEP 2079 H=2079:00 +CPU= 35.041
  15:44:09 STEP 2080 H=2080:00 +CPU= 23.334
  15:44:34 STEP 2081 H=2081:00 +CPU= 22.595
  15:45:14 STEP 2082 H=2082:00 +CPU= 35.729
  15:45:39 STEP 2083 H=2083:00 +CPU= 22.685
  15:46:05 STEP 2084 H=2084:00 +CPU= 23.226
  15:46:43 STEP 2085 H=2085:00 +CPU= 35.045
  15:47:09 STEP 2086 H=2086:00 +CPU= 23.293
  15:47:35 STEP 2087 H=2087:00 +CPU= 23.198
  15:48:21 STEP 2088 H=2088:00 +CPU= 41.557
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_86.zip
Uploading the intermediate file: upload_file_86.zip
  15:48:47 STEP 2089 H=2089:00 +CPU= 22.631
  15:49:13 STEP 2090 H=2090:00 +CPU= 23.395
  15:49:51 STEP 2091 H=2091:00 +CPU= 34.830
  15:50:17 STEP 2092 H=2092:00 +CPU= 23.535
  15:50:42 STEP 2093 H=2093:00 +CPU= 22.678
  15:51:21 STEP 2094 H=2094:00 +CPU= 35.433
  15:51:46 STEP 2095 H=2095:00 +CPU= 22.551
  15:52:11 STEP 2096 H=2096:00 +CPU= 23.281
  15:52:50 STEP 2097 H=2097:00 +CPU= 34.876
  15:53:16 STEP 2098 H=2098:00 +CPU= 23.411
  15:53:41 STEP 2099 H=2099:00 +CPU= 22.579
  15:54:24 STEP 2100 H=2100:00 +CPU= 38.877
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002100
  15:54:49 STEP 2101 H=2101:00 +CPU= 22.853
  15:55:15 STEP 2102 H=2102:00 +CPU= 23.322
  15:55:54 STEP 2103 H=2103:00 +CPU= 34.749
  15:56:20 STEP 2104 H=2104:00 +CPU= 23.158
  15:56:45 STEP 2105 H=2105:00 +CPU= 22.598
  15:57:24 STEP 2106 H=2106:00 +CPU= 35.725
  15:57:50 STEP 2107 H=2107:00 +CPU= 22.719
  15:58:16 STEP 2108 H=2108:00 +CPU= 23.506
  15:58:54 STEP 2109 H=2109:00 +CPU= 34.747
  15:59:20 STEP 2110 H=2110:00 +CPU= 23.153
  15:59:46 STEP 2111 H=2111:00 +CPU= 23.522
  16:00:32 STEP 2112 H=2112:00 +CPU= 41.513
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_87.zip
Uploading the intermediate file: upload_file_87.zip
  16:00:57 STEP 2113 H=2113:00 +CPU= 22.543
  16:01:23 STEP 2114 H=2114:00 +CPU= 23.412
  16:02:03 STEP 2115 H=2115:00 +CPU= 35.007
  16:02:28 STEP 2116 H=2116:00 +CPU= 23.361
  16:02:53 STEP 2117 H=2117:00 +CPU= 22.575
  16:03:33 STEP 2118 H=2118:00 +CPU= 35.687
  16:03:58 STEP 2119 H=2119:00 +CPU= 22.547
  16:04:23 STEP 2120 H=2120:00 +CPU= 23.241
  16:05:02 STEP 2121 H=2121:00 +CPU= 34.792
  16:05:31 STEP 2122 H=2122:00 +CPU= 23.749
  16:05:56 STEP 2123 H=2123:00 +CPU= 22.640
  16:06:39 STEP 2124 H=2124:00 +CPU= 38.855
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002124
  16:07:04 STEP 2125 H=2125:00 +CPU= 22.545
  16:07:30 STEP 2126 H=2126:00 +CPU= 23.456
  16:08:09 STEP 2127 H=2127:00 +CPU= 34.837
  16:08:35 STEP 2128 H=2128:00 +CPU= 23.294
  16:09:00 STEP 2129 H=2129:00 +CPU= 22.624
  16:09:39 STEP 2130 H=2130:00 +CPU= 35.714
  16:10:04 STEP 2131 H=2131:00 +CPU= 22.819
  16:10:30 STEP 2132 H=2132:00 +CPU= 23.265
  16:11:09 STEP 2133 H=2133:00 +CPU= 35.123
  16:11:35 STEP 2134 H=2134:00 +CPU= 23.240
  16:12:01 STEP 2135 H=2135:00 +CPU= 23.648
  16:12:47 STEP 2136 H=2136:00 +CPU= 41.445
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_88.zip
Uploading the intermediate file: upload_file_88.zip
  16:13:14 STEP 2137 H=2137:00 +CPU= 22.700
  16:13:40 STEP 2138 H=2138:00 +CPU= 23.315
  16:14:18 STEP 2139 H=2139:00 +CPU= 34.813
  16:14:44 STEP 2140 H=2140:00 +CPU= 23.331
  16:15:12 STEP 2141 H=2141:00 +CPU= 22.914
  16:15:53 STEP 2142 H=2142:00 +CPU= 35.826
  16:16:19 STEP 2143 H=2143:00 +CPU= 22.802
  16:16:55 STEP 2144 H=2144:00 +CPU= 24.083
  16:17:36 STEP 2145 H=2145:00 +CPU= 35.133
  16:18:02 STEP 2146 H=2146:00 +CPU= 23.441
  16:18:27 STEP 2147 H=2147:00 +CPU= 22.858
  16:19:11 STEP 2148 H=2148:00 +CPU= 39.045
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002148
  16:19:36 STEP 2149 H=2149:00 +CPU= 22.687
  16:20:02 STEP 2150 H=2150:00 +CPU= 23.364
  16:20:41 STEP 2151 H=2151:00 +CPU= 34.725
  16:21:06 STEP 2152 H=2152:00 +CPU= 23.390
  16:21:34 STEP 2153 H=2153:00 +CPU= 22.994
  16:22:14 STEP 2154 H=2154:00 +CPU= 35.906
  16:22:39 STEP 2155 H=2155:00 +CPU= 23.096
  16:23:05 STEP 2156 H=2156:00 +CPU= 23.728
  16:23:44 STEP 2157 H=2157:00 +CPU= 35.128
  16:24:10 STEP 2158 H=2158:00 +CPU= 23.582
  16:24:36 STEP 2159 H=2159:00 +CPU= 23.786
  16:25:23 STEP 2160 H=2160:00 +CPU= 41.783
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_89.zip
Uploading the intermediate file: upload_file_89.zip
  16:25:49 STEP 2161 H=2161:00 +CPU= 22.706
  16:26:16 STEP 2162 H=2162:00 +CPU= 23.715
  16:26:56 STEP 2163 H=2163:00 +CPU= 35.524
  16:27:23 STEP 2164 H=2164:00 +CPU= 23.505
  16:27:48 STEP 2165 H=2165:00 +CPU= 22.794
  16:28:28 STEP 2166 H=2166:00 +CPU= 35.852
  16:28:54 STEP 2167 H=2167:00 +CPU= 22.845
  16:29:21 STEP 2168 H=2168:00 +CPU= 23.568
  16:30:01 STEP 2169 H=2169:00 +CPU= 35.348
  16:30:28 STEP 2170 H=2170:00 +CPU= 23.556
  16:30:54 STEP 2171 H=2171:00 +CPU= 22.798
  16:31:41 STEP 2172 H=2172:00 +CPU= 39.495
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002172
  16:32:04 STEP 2173 H=2173:00 +CPU= 21.436
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0476
(argv4) batchid: 974
(argv5) wuid: 12191120
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f0857570076b574500063f8b6c500254 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0476_2005050100_123_974_12191120.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0476_2005050100_123_974_12191120.zip
ic_ancil_file: ic_ancil_12191120
ifsdata_file: ifsdata_12191120
climate_data_file: clim_data_12191120
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0fee53bf0baeaf8a2a1e9a6c746c973 to: /var/lib/boinc-client/slots/3/ic_ancil_12191120.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12191120.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_909ea19238b177560884a218dca03496 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191120.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191120.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191120.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191120.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12191120.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12191120.xml
-- Model is restarting --
Adjusting last_iter, 2174, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12191120.xml
last_cpu_time: 63148
upload_file_number: 90
last_iter: 2159
last_upload: 7776000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175
The child process has been launched with process id: 9280
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:naturerick:1:1:9280:9280] [20230102:111426:1672686866:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:15:25 STEP 2160 H=2160:00 +CPU= 40.757
  11:15:54 STEP 2161 H=2161:00 +CPU= 22.619
  11:16:21 STEP 2162 H=2162:00 +CPU= 23.195
  11:17:03 STEP 2163 H=2163:00 +CPU= 34.948
  11:17:28 STEP 2164 H=2164:00 +CPU= 22.940
  11:17:55 STEP 2165 H=2165:00 +CPU= 22.465
  11:18:38 STEP 2166 H=2166:00 +CPU= 36.070
  11:19:05 STEP 2167 H=2167:00 +CPU= 22.652
  11:19:31 STEP 2168 H=2168:00 +CPU= 23.049
  11:20:10 STEP 2169 H=2169:00 +CPU= 34.460
  11:20:38 STEP 2170 H=2170:00 +CPU= 23.149
  11:21:04 STEP 2171 H=2171:00 +CPU= 22.343
  11:21:48 STEP 2172 H=2172:00 +CPU= 38.349
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002172
  11:22:13 STEP 2173 H=2173:00 +CPU= 22.341
  11:22:39 STEP 2174 H=2174:00 +CPU= 23.184
  11:23:27 STEP 2175 H=2175:00 +CPU= 35.671
  11:23:58 STEP 2176 H=2176:00 +CPU= 23.657
  11:24:25 STEP 2177 H=2177:00 +CPU= 22.802
  11:25:18 STEP 2178 H=2178:00 +CPU= 36.900
  11:25:45 STEP 2179 H=2179:00 +CPU= 22.935
  11:26:14 STEP 2180 H=2180:00 +CPU= 23.530
  11:26:59 STEP 2181 H=2181:00 +CPU= 35.674
  11:27:27 STEP 2182 H=2182:00 +CPU= 23.347
  11:27:55 STEP 2183 H=2183:00 +CPU= 23.432
  11:28:43 STEP 2184 H=2184:00 +CPU= 41.386
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191120/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0476_2005050100_123_974_12191120_0_r1858008175_90.zip
Uploading the intermediate file: upload_file_90.zip
  11:29:09 STEP 2185 H=2185:00 +CPU= 22.667
  11:29:35 STEP 2186 H=2186:00 +CPU= 23.264
  11:30:14 STEP 2187 H=2187:00 +CPU= 34.654
  11:30:40 STEP 2188 H=2188:00 +CPU= 23.219
  11:31:05 STEP 2189 H=2189:00 +CPU= 22.540
  11:31:45 STEP 2190 H=2190:00 +CPU= 35.695
  11:32:11 STEP 2191 H=2191:00 +CPU= 22.603
  11:32:37 STEP 2192 H=2192:00 +CPU= 23.097
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:27 1506016 22276783 oifs_43r3_ps_0476_2005050100_123_974_12191120_0 6,998,400 56,843 0.0081
09 May 2023 16:04:49 1506016 22276783 oifs_43r3_ps_0476_2005050100_123_974_12191120_0 5,616,000 45,681 0.0081
09 May 2023 16:04:39 1506016 22276783 oifs_43r3_ps_0476_2005050100_123_974_12191120_0 4,320,000 34,727 0.0080
09 May 2023 16:04:37 1506016 22276783 oifs_43r3_ps_0476_2005050100_123_974_12191120_0 3,456,000 27,981 0.0081
09 May 2023 16:03:29 1506016 22276783 oifs_43r3_ps_0476_2005050100_123_974_12191120_0 1,468,800 11,670 0.0079


©2024 climateprediction.net