climateprediction.net home page
Task 22247736

Task 22247736

Name oifs_43r3_ps_2622_2021050100_123_947_12165711_0
Workunit 12165711
Created 29 Nov 2022, 14:26:21 UTC
Sent 29 Nov 2022, 15:46:29 UTC
Report deadline 29 Dec 2022, 15:46:29 UTC
Received 30 Nov 2022, 4:37:36 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1536641
Run time 11 hours 57 min 51 sec
CPU time 2 hours 8 min 31 sec
Validate state Invalid
Credit 10,281.60
Device peak FLOPS 5.08 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,411.87 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,204.62 MB
Stderr
<core_client_version>7.20.4</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
net/oifs_43r3_ps_12165711/ICMUAhpi1+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_65.zip
Uploading the intermediate file: upload_file_65.zip
  16:47:29 STEP 1585 H=1585:00 +CPU= 16.308
Uploading trickle at timestep: 5702400
  16:47:47 STEP 1586 H=1586:00 +CPU= 16.747
  16:48:13 STEP 1587 H=1587:00 +CPU= 25.434
  16:48:29 STEP 1588 H=1588:00 +CPU= 16.278
  16:48:45 STEP 1589 H=1589:00 +CPU= 16.158
  16:49:12 STEP 1590 H=1590:00 +CPU= 26.098
  16:49:28 STEP 1591 H=1591:00 +CPU= 15.878
  16:49:44 STEP 1592 H=1592:00 +CPU= 16.634
  16:50:10 STEP 1593 H=1593:00 +CPU= 25.487
  16:50:26 STEP 1594 H=1594:00 +CPU= 16.245
  16:50:43 STEP 1595 H=1595:00 +CPU= 16.155
  16:51:11 STEP 1596 H=1596:00 +CPU= 28.042
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001596
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001596
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001596
  16:51:27 STEP 1597 H=1597:00 +CPU= 15.885
  16:51:44 STEP 1598 H=1598:00 +CPU= 16.587
  16:52:12 STEP 1599 H=1599:00 +CPU= 25.319
  16:52:30 STEP 1600 H=1600:00 +CPU= 16.467
  16:52:47 STEP 1601 H=1601:00 +CPU= 16.159
  16:53:13 STEP 1602 H=1602:00 +CPU= 25.835
  16:53:29 STEP 1603 H=1603:00 +CPU= 16.017
  16:53:46 STEP 1604 H=1604:00 +CPU= 16.585
  16:54:11 STEP 1605 H=1605:00 +CPU= 25.290
  16:54:28 STEP 1606 H=1606:00 +CPU= 16.439
  16:54:44 STEP 1607 H=1607:00 +CPU= 16.593
  16:55:17 STEP 1608 H=1608:00 +CPU= 29.720
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001608
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001608
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_66.zip
Uploading the intermediate file: upload_file_66.zip
  16:55:33 STEP 1609 H=1609:00 +CPU= 15.817
Uploading trickle at timestep: 5788800
  16:55:49 STEP 1610 H=1610:00 +CPU= 16.663
  16:56:15 STEP 1611 H=1611:00 +CPU= 25.441
  16:56:32 STEP 1612 H=1612:00 +CPU= 16.389
  16:56:48 STEP 1613 H=1613:00 +CPU= 16.132
  16:57:17 STEP 1614 H=1614:00 +CPU= 26.266
  16:57:33 STEP 1615 H=1615:00 +CPU= 15.912
  16:57:50 STEP 1616 H=1616:00 +CPU= 16.686
  16:58:20 STEP 1617 H=1617:00 +CPU= 25.830
  16:58:37 STEP 1618 H=1618:00 +CPU= 16.507
  16:58:54 STEP 1619 H=1619:00 +CPU= 16.220
  16:59:22 STEP 1620 H=1620:00 +CPU= 28.151
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001620
  16:59:38 STEP 1621 H=1621:00 +CPU= 15.924
  16:59:55 STEP 1622 H=1622:00 +CPU= 16.686
  17:00:21 STEP 1623 H=1623:00 +CPU= 25.605
  17:00:36 STEP 1624 H=1624:00 +CPU= 15.587
  17:00:52 STEP 1625 H=1625:00 +CPU= 15.938
  17:01:20 STEP 1626 H=1626:00 +CPU= 26.060
  17:01:39 STEP 1627 H=1627:00 +CPU= 16.263
  17:01:56 STEP 1628 H=1628:00 +CPU= 16.452
  17:02:23 STEP 1629 H=1629:00 +CPU= 25.505
  17:02:40 STEP 1630 H=1630:00 +CPU= 16.614
  17:02:56 STEP 1631 H=1631:00 +CPU= 16.414
  17:03:36 STEP 1632 H=1632:00 +CPU= 29.693
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_67.zip
Uploading the intermediate file: upload_file_67.zip
  17:03:53 STEP 1633 H=1633:00 +CPU= 16.168
Uploading trickle at timestep: 5875200
  17:04:09 STEP 1634 H=1634:00 +CPU= 16.688
  17:04:35 STEP 1635 H=1635:00 +CPU= 25.292
  17:04:52 STEP 1636 H=1636:00 +CPU= 16.636
  17:05:08 STEP 1637 H=1637:00 +CPU= 15.907
  17:05:34 STEP 1638 H=1638:00 +CPU= 25.764
  17:05:50 STEP 1639 H=1639:00 +CPU= 16.100
  17:06:25 STEP 1640 H=1640:00 +CPU= 16.857
  17:06:50 STEP 1641 H=1641:00 +CPU= 25.495
  17:07:07 STEP 1642 H=1642:00 +CPU= 16.665
  17:07:23 STEP 1643 H=1643:00 +CPU= 15.964
  17:07:52 STEP 1644 H=1644:00 +CPU= 27.987
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001644
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001644
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001644
  17:08:08 STEP 1645 H=1645:00 +CPU= 16.080
  17:08:25 STEP 1646 H=1646:00 +CPU= 16.565
  17:08:50 STEP 1647 H=1647:00 +CPU= 25.343
  17:09:08 STEP 1648 H=1648:00 +CPU= 16.375
  17:09:24 STEP 1649 H=1649:00 +CPU= 15.909
  17:09:50 STEP 1650 H=1650:00 +CPU= 26.015
  17:10:06 STEP 1651 H=1651:00 +CPU= 16.124
  17:10:23 STEP 1652 H=1652:00 +CPU= 16.391
  17:10:48 STEP 1653 H=1653:00 +CPU= 25.465
  17:11:05 STEP 1654 H=1654:00 +CPU= 16.627
  17:11:22 STEP 1655 H=1655:00 +CPU= 16.388
  17:11:58 STEP 1656 H=1656:00 +CPU= 29.721
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001656
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001656
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_68.zip
Uploading the intermediate file: upload_file_68.zip
  17:12:14 STEP 1657 H=1657:00 +CPU= 16.147
Uploading trickle at timestep: 5961600
  17:12:33 STEP 1658 H=1658:00 +CPU= 16.469
  17:12:59 STEP 1659 H=1659:00 +CPU= 25.369
  17:13:16 STEP 1660 H=1660:00 +CPU= 16.595
  17:13:32 STEP 1661 H=1661:00 +CPU= 15.760
  17:13:58 STEP 1662 H=1662:00 +CPU= 26.095
  17:14:14 STEP 1663 H=1663:00 +CPU= 16.034
  17:14:30 STEP 1664 H=1664:00 +CPU= 16.274
  17:14:56 STEP 1665 H=1665:00 +CPU= 25.387
  17:15:13 STEP 1666 H=1666:00 +CPU= 16.590
  17:15:29 STEP 1667 H=1667:00 +CPU= 15.811
  17:15:57 STEP 1668 H=1668:00 +CPU= 28.144
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001668
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001668
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001668
  17:16:13 STEP 1669 H=1669:00 +CPU= 16.149
  17:16:32 STEP 1670 H=1670:00 +CPU= 16.712
  17:16:58 STEP 1671 H=1671:00 +CPU= 25.386
  17:17:15 STEP 1672 H=1672:00 +CPU= 16.369
  17:17:31 STEP 1673 H=1673:00 +CPU= 15.977
  17:17:57 STEP 1674 H=1674:00 +CPU= 25.950
  17:18:14 STEP 1675 H=1675:00 +CPU= 16.013
  17:18:30 STEP 1676 H=1676:00 +CPU= 16.464
  17:18:56 STEP 1677 H=1677:00 +CPU= 25.347
  17:19:12 STEP 1678 H=1678:00 +CPU= 16.490
  17:19:29 STEP 1679 H=1679:00 +CPU= 16.371
  17:20:05 STEP 1680 H=1680:00 +CPU= 29.720
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001680
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001680
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_69.zip
Uploading the intermediate file: upload_file_69.zip
  17:20:21 STEP 1681 H=1681:00 +CPU= 16.171
Uploading trickle at timestep: 6048000
  17:20:40 STEP 1682 H=1682:00 +CPU= 16.362
  17:21:05 STEP 1683 H=1683:00 +CPU= 25.443
  17:21:22 STEP 1684 H=1684:00 +CPU= 16.632
  17:21:38 STEP 1685 H=1685:00 +CPU= 15.772
  17:22:04 STEP 1686 H=1686:00 +CPU= 26.172
  17:22:21 STEP 1687 H=1687:00 +CPU= 16.179
  17:22:37 STEP 1688 H=1688:00 +CPU= 16.333
  17:23:03 STEP 1689 H=1689:00 +CPU= 25.592
  17:23:20 STEP 1690 H=1690:00 +CPU= 16.671
  17:23:36 STEP 1691 H=1691:00 +CPU= 15.852
  17:24:04 STEP 1692 H=1692:00 +CPU= 28.209
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001692
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001692
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001692
  17:24:20 STEP 1693 H=1693:00 +CPU= 16.167
  17:24:37 STEP 1694 H=1694:00 +CPU= 16.310
  17:25:03 STEP 1695 H=1695:00 +CPU= 25.505
  17:25:19 STEP 1696 H=1696:00 +CPU= 16.513
  17:25:35 STEP 1697 H=1697:00 +CPU= 15.977
  17:26:12 STEP 1698 H=1698:00 +CPU= 26.308
  17:26:31 STEP 1699 H=1699:00 +CPU= 16.369
  17:26:47 STEP 1700 H=1700:00 +CPU= 16.467
  17:27:14 STEP 1701 H=1701:00 +CPU= 25.610
  17:27:31 STEP 1702 H=1702:00 +CPU= 16.725
  17:27:48 STEP 1703 H=1703:00 +CPU= 16.477
  17:28:31 STEP 1704 H=1704:00 +CPU= 29.775
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_70.zip
Uploading the intermediate file: upload_file_70.zip
  17:28:47 STEP 1705 H=1705:00 +CPU= 16.190
Uploading trickle at timestep: 6134400
  17:29:05 STEP 1706 H=1706:00 +CPU= 16.696
  17:29:30 STEP 1707 H=1707:00 +CPU= 25.417
  17:29:47 STEP 1708 H=1708:00 +CPU= 16.655
  17:30:03 STEP 1709 H=1709:00 +CPU= 15.877
  17:30:29 STEP 1710 H=1710:00 +CPU= 26.121
  17:30:46 STEP 1711 H=1711:00 +CPU= 16.225
  17:31:02 STEP 1712 H=1712:00 +CPU= 16.420
  17:31:28 STEP 1713 H=1713:00 +CPU= 25.569
  17:31:45 STEP 1714 H=1714:00 +CPU= 16.667
  17:32:01 STEP 1715 H=1715:00 +CPU= 15.956
  17:32:30 STEP 1716 H=1716:00 +CPU= 28.291
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001716
  17:32:46 STEP 1717 H=1717:00 +CPU= 16.180
  17:33:02 STEP 1718 H=1718:00 +CPU= 16.458
  17:33:28 STEP 1719 H=1719:00 +CPU= 25.637
  17:33:45 STEP 1720 H=1720:00 +CPU= 16.698
  17:34:01 STEP 1721 H=1721:00 +CPU= 15.837
  17:34:30 STEP 1722 H=1722:00 +CPU= 26.129
  17:34:47 STEP 1723 H=1723:00 +CPU= 16.513
  17:35:04 STEP 1724 H=1724:00 +CPU= 16.934
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.395] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4031MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.396] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.396] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.396] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.396] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.396] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4031 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.398] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.401] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.402] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.402] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173511:1669772111:35297.407] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_mcica_sw:solver_mcica_sw 
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173512:1669772112:35298.407] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:l-quasar:1:1:2424:2424] [20221129:173512:1669772112:35298.407] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
master.exe         0000000001CE916B  Unknown               Unknown  Unknown
master.exe         000000000081FFF1  Unknown               Unknown  Unknown
master.exe         0000000001DC9090  Unknown               Unknown  Unknown
master.exe         0000000001CC8D8C  Unknown               Unknown  Unknown
master.exe         0000000001CC0C9D  Unknown               Unknown  Unknown
master.exe         000000000188C190  radiation_two_str         384  radiation_two_stream.F90
master.exe         0000000001968BFA  radiation_mcica_s         172  radiation_mcica_sw.F90
master.exe         0000000001862DC8  radiation_interfa         419  radiation_interface.F90
master.exe         00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
master.exe         000000000129A415  radintg_                 1455  radintg.F90
master.exe         0000000001270124  raddrv_                   537  raddrv.F90
master.exe         0000000000BFD839  gp_model_                 482  gp_model.F90
master.exe         00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
master.exe         0000000000BBC1CE  scan2m_                   535  scan2m.F90
master.exe         000000000057A664  stepo_                    327  stepo.F90
master.exe         000000000055DEEC  cnt4_                    1133  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 2622
(argv4) batchid: 947
(argv5) wuid: 12165711
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_199888e14664aa116214256bab164e38 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_2622_2021050100_123_947_12165711.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_2622_2021050100_123_947_12165711.zip
ic_ancil_file: ic_ancil_12165711
ifsdata_file: ifsdata_12165711
climate_data_file: clim_data_12165711
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_39d18f5b2b1cd9b509c36b8c95beecfd to: /var/lib/boinc-client/slots/2/ic_ancil_12165711.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12165711.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_65b4deaabb9e7c159c037acfd4a32c4b to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12165711.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12165711.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12165711.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12165711.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110
The child process has been launched with process id: 1633
Executing the command: /var/lib/boinc-client/slots/2/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:l-quasar:1:1:1633:1633] [20221129:174529:1669772729:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:46:14 STEP 1704 H=1704:00 +CPU= 28.880
  17:46:30 STEP 1705 H=1705:00 +CPU= 16.289
  17:46:47 STEP 1706 H=1706:00 +CPU= 16.633
  17:47:13 STEP 1707 H=1707:00 +CPU= 26.098
  17:47:32 STEP 1708 H=1708:00 +CPU= 18.079
  17:47:50 STEP 1709 H=1709:00 +CPU= 18.135
  17:48:19 STEP 1710 H=1710:00 +CPU= 28.628
  17:48:38 STEP 1711 H=1711:00 +CPU= 17.832
  17:48:56 STEP 1712 H=1712:00 +CPU= 18.275
  17:49:24 STEP 1713 H=1713:00 +CPU= 27.637
  17:49:42 STEP 1714 H=1714:00 +CPU= 18.133
  17:50:00 STEP 1715 H=1715:00 +CPU= 17.809
  17:50:31 STEP 1716 H=1716:00 +CPU= 30.578
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001716
  17:50:49 STEP 1717 H=1717:00 +CPU= 17.531
  17:51:07 STEP 1718 H=1718:00 +CPU= 18.428
  17:51:35 STEP 1719 H=1719:00 +CPU= 27.764
  17:51:54 STEP 1720 H=1720:00 +CPU= 18.188
  17:52:12 STEP 1721 H=1721:00 +CPU= 17.919
  17:52:40 STEP 1722 H=1722:00 +CPU= 28.216
  17:52:58 STEP 1723 H=1723:00 +CPU= 17.681
  17:53:17 STEP 1724 H=1724:00 +CPU= 18.300
  17:53:45 STEP 1725 H=1725:00 +CPU= 27.767
  17:54:03 STEP 1726 H=1726:00 +CPU= 18.126
  17:54:21 STEP 1727 H=1727:00 +CPU= 18.391
  17:54:56 STEP 1728 H=1728:00 +CPU= 32.397
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001728
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001728
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_1.zip
Uploading the intermediate file: upload_file_1.zip
  17:55:14 STEP 1729 H=1729:00 +CPU= 17.514
Uploading trickle at timestep: 6220800
  17:55:32 STEP 1730 H=1730:00 +CPU= 18.568
  17:56:00 STEP 1731 H=1731:00 +CPU= 27.713
  17:56:19 STEP 1732 H=1732:00 +CPU= 18.164
  17:56:37 STEP 1733 H=1733:00 +CPU= 17.869
  17:57:05 STEP 1734 H=1734:00 +CPU= 28.348
  17:57:23 STEP 1735 H=1735:00 +CPU= 17.774
  17:57:42 STEP 1736 H=1736:00 +CPU= 18.452
  17:58:10 STEP 1737 H=1737:00 +CPU= 27.823
  17:58:28 STEP 1738 H=1738:00 +CPU= 18.212
  17:58:46 STEP 1739 H=1739:00 +CPU= 17.991
  17:59:17 STEP 1740 H=1740:00 +CPU= 30.761
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001740
  17:59:35 STEP 1741 H=1741:00 +CPU= 17.540
  17:59:54 STEP 1742 H=1742:00 +CPU= 18.343
  18:00:22 STEP 1743 H=1743:00 +CPU= 27.871
  18:00:40 STEP 1744 H=1744:00 +CPU= 17.988
  18:00:58 STEP 1745 H=1745:00 +CPU= 17.880
  18:01:26 STEP 1746 H=1746:00 +CPU= 28.300
  18:01:44 STEP 1747 H=1747:00 +CPU= 17.487
  18:02:02 STEP 1748 H=1748:00 +CPU= 18.350
  18:02:31 STEP 1749 H=1749:00 +CPU= 27.802
  18:02:49 STEP 1750 H=1750:00 +CPU= 17.970
  18:03:07 STEP 1751 H=1751:00 +CPU= 18.374
  18:03:40 STEP 1752 H=1752:00 +CPU= 32.268
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001752
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001752
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_2.zip
Uploading the intermediate file: upload_file_2.zip
  18:03:58 STEP 1753 H=1753:00 +CPU= 17.581
Uploading trickle at timestep: 6307200
  18:04:16 STEP 1754 H=1754:00 +CPU= 18.307
  18:04:44 STEP 1755 H=1755:00 +CPU= 27.648
  18:05:02 STEP 1756 H=1756:00 +CPU= 18.009
  18:05:20 STEP 1757 H=1757:00 +CPU= 17.984
  18:05:49 STEP 1758 H=1758:00 +CPU= 28.399
  18:06:07 STEP 1759 H=1759:00 +CPU= 17.445
  18:06:25 STEP 1760 H=1760:00 +CPU= 18.335
  18:06:53 STEP 1761 H=1761:00 +CPU= 27.900
  18:07:11 STEP 1762 H=1762:00 +CPU= 17.970
  18:07:29 STEP 1763 H=1763:00 +CPU= 17.862
  18:08:00 STEP 1764 H=1764:00 +CPU= 30.620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001764
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001764
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001764
  18:08:18 STEP 1765 H=1765:00 +CPU= 17.593
  18:08:37 STEP 1766 H=1766:00 +CPU= 18.430
  18:09:05 STEP 1767 H=1767:00 +CPU= 27.932
  18:09:23 STEP 1768 H=1768:00 +CPU= 17.968
  18:09:41 STEP 1769 H=1769:00 +CPU= 17.840
  18:10:10 STEP 1770 H=1770:00 +CPU= 28.512
  18:10:27 STEP 1771 H=1771:00 +CPU= 17.501
  18:10:46 STEP 1772 H=1772:00 +CPU= 18.353
  18:11:14 STEP 1773 H=1773:00 +CPU= 27.951
  18:11:32 STEP 1774 H=1774:00 +CPU= 17.978
  18:11:51 STEP 1775 H=1775:00 +CPU= 18.424
  18:12:24 STEP 1776 H=1776:00 +CPU= 32.255
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001776
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001776
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_3.zip
Uploading the intermediate file: upload_file_3.zip
  18:12:42 STEP 1777 H=1777:00 +CPU= 17.839
Uploading trickle at timestep: 6393600
  18:13:01 STEP 1778 H=1778:00 +CPU= 18.514
  18:13:29 STEP 1779 H=1779:00 +CPU= 27.850
  18:13:48 STEP 1780 H=1780:00 +CPU= 18.026
  18:14:06 STEP 1781 H=1781:00 +CPU= 17.991
  18:14:34 STEP 1782 H=1782:00 +CPU= 28.469
  18:14:52 STEP 1783 H=1783:00 +CPU= 17.559
  18:15:11 STEP 1784 H=1784:00 +CPU= 18.439
  18:15:39 STEP 1785 H=1785:00 +CPU= 27.869
  18:15:57 STEP 1786 H=1786:00 +CPU= 18.065
  18:16:15 STEP 1787 H=1787:00 +CPU= 17.936
  18:16:46 STEP 1788 H=1788:00 +CPU= 30.603
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001788
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001788
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001788
  18:17:04 STEP 1789 H=1789:00 +CPU= 17.764
  18:17:23 STEP 1790 H=1790:00 +CPU= 18.462
  18:17:51 STEP 1791 H=1791:00 +CPU= 27.838
  18:18:09 STEP 1792 H=1792:00 +CPU= 18.164
  18:18:27 STEP 1793 H=1793:00 +CPU= 17.998
  18:18:56 STEP 1794 H=1794:00 +CPU= 28.295
  18:19:14 STEP 1795 H=1795:00 +CPU= 17.772
  18:19:32 STEP 1796 H=1796:00 +CPU= 18.346
  18:20:00 STEP 1797 H=1797:00 +CPU= 27.696
  18:20:19 STEP 1798 H=1798:00 +CPU= 18.197
  18:20:37 STEP 1799 H=1799:00 +CPU= 18.488
  18:21:09 STEP 1800 H=1800:00 +CPU= 31.863
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_4.zip
Uploading the intermediate file: upload_file_4.zip
  18:21:27 STEP 1801 H=1801:00 +CPU= 17.860
Uploading trickle at timestep: 6480000
  18:21:46 STEP 1802 H=1802:00 +CPU= 18.334
  18:22:14 STEP 1803 H=1803:00 +CPU= 27.540
  18:22:32 STEP 1804 H=1804:00 +CPU= 18.176
  18:22:50 STEP 1805 H=1805:00 +CPU= 17.938
  18:23:19 STEP 1806 H=1806:00 +CPU= 28.289
  18:23:37 STEP 1807 H=1807:00 +CPU= 17.744
  18:23:55 STEP 1808 H=1808:00 +CPU= 18.394
  18:24:23 STEP 1809 H=1809:00 +CPU= 27.819
  18:24:41 STEP 1810 H=1810:00 +CPU= 18.262
  18:25:00 STEP 1811 H=1811:00 +CPU= 18.053
  18:25:30 STEP 1812 H=1812:00 +CPU= 30.397
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001812
  18:25:49 STEP 1813 H=1813:00 +CPU= 17.879
  18:26:07 STEP 1814 H=1814:00 +CPU= 18.328
  18:26:35 STEP 1815 H=1815:00 +CPU= 27.580
  18:26:53 STEP 1816 H=1816:00 +CPU= 18.256
  18:27:11 STEP 1817 H=1817:00 +CPU= 17.910
  18:27:40 STEP 1818 H=1818:00 +CPU= 28.127
  18:27:58 STEP 1819 H=1819:00 +CPU= 17.752
  18:28:16 STEP 1820 H=1820:00 +CPU= 18.330
  18:28:44 STEP 1821 H=1821:00 +CPU= 27.562
  18:29:02 STEP 1822 H=1822:00 +CPU= 18.266
  18:29:21 STEP 1823 H=1823:00 +CPU= 18.474
  18:29:57 STEP 1824 H=1824:00 +CPU= 31.916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_5.zip
Uploading the intermediate file: upload_file_5.zip
  18:30:15 STEP 1825 H=1825:00 +CPU= 17.869
Uploading trickle at timestep: 6566400
  18:30:33 STEP 1826 H=1826:00 +CPU= 18.414
  18:31:01 STEP 1827 H=1827:00 +CPU= 27.468
  18:31:20 STEP 1828 H=1828:00 +CPU= 18.487
  18:31:38 STEP 1829 H=1829:00 +CPU= 17.991
  18:32:06 STEP 1830 H=1830:00 +CPU= 28.121
  18:32:24 STEP 1831 H=1831:00 +CPU= 17.988
  18:32:43 STEP 1832 H=1832:00 +CPU= 18.408
  18:33:11 STEP 1833 H=1833:00 +CPU= 27.612
  18:33:29 STEP 1834 H=1834:00 +CPU= 18.436
  18:33:47 STEP 1835 H=1835:00 +CPU= 17.973
  18:34:18 STEP 1836 H=1836:00 +CPU= 30.386
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001836
  18:34:36 STEP 1837 H=1837:00 +CPU= 18.010
  18:34:55 STEP 1838 H=1838:00 +CPU= 18.413
  18:35:23 STEP 1839 H=1839:00 +CPU= 27.586
  18:35:41 STEP 1840 H=1840:00 +CPU= 18.425
  18:35:59 STEP 1841 H=1841:00 +CPU= 17.936
  18:36:28 STEP 1842 H=1842:00 +CPU= 28.040
  18:36:46 STEP 1843 H=1843:00 +CPU= 17.909
  18:37:04 STEP 1844 H=1844:00 +CPU= 18.356
  18:37:32 STEP 1845 H=1845:00 +CPU= 27.474
  18:37:50 STEP 1846 H=1846:00 +CPU= 18.355
  18:38:09 STEP 1847 H=1847:00 +CPU= 18.399
  18:38:48 STEP 1848 H=1848:00 +CPU= 32.007
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_6.zip
Uploading the intermediate file: upload_file_6.zip
  18:39:06 STEP 1849 H=1849:00 +CPU= 17.744
Uploading trickle at timestep: 6652800
  18:39:25 STEP 1850 H=1850:00 +CPU= 18.303
  18:39:52 STEP 1851 H=1851:00 +CPU= 27.425
  18:40:11 STEP 1852 H=1852:00 +CPU= 18.469
  18:40:29 STEP 1853 H=1853:00 +CPU= 17.929
  18:40:58 STEP 1854 H=1854:00 +CPU= 28.062
  18:41:16 STEP 1855 H=1855:00 +CPU= 17.883
  18:41:34 STEP 1856 H=1856:00 +CPU= 18.424
  18:42:02 STEP 1857 H=1857:00 +CPU= 27.606
  18:42:21 STEP 1858 H=1858:00 +CPU= 18.382
  18:42:39 STEP 1859 H=1859:00 +CPU= 17.936
  18:43:09 STEP 1860 H=1860:00 +CPU= 30.387
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001860
  18:43:27 STEP 1861 H=1861:00 +CPU= 17.857
  18:43:46 STEP 1862 H=1862:00 +CPU= 18.460
  18:44:14 STEP 1863 H=1863:00 +CPU= 27.524
  18:44:32 STEP 1864 H=1864:00 +CPU= 18.340
  18:44:50 STEP 1865 H=1865:00 +CPU= 17.912
  18:45:18 STEP 1866 H=1866:00 +CPU= 28.050
  18:45:36 STEP 1867 H=1867:00 +CPU= 17.868
  18:45:55 STEP 1868 H=1868:00 +CPU= 18.429
  18:46:23 STEP 1869 H=1869:00 +CPU= 27.751
  18:46:42 STEP 1870 H=1870:00 +CPU= 18.371
  18:47:00 STEP 1871 H=1871:00 +CPU= 18.409
  18:47:35 STEP 1872 H=1872:00 +CPU= 31.783
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_7.zip
Uploading the intermediate file: upload_file_7.zip
  18:47:53 STEP 1873 H=1873:00 +CPU= 17.818
Uploading trickle at timestep: 6739200
  18:48:12 STEP 1874 H=1874:00 +CPU= 18.362
  18:48:39 STEP 1875 H=1875:00 +CPU= 27.459
  18:48:58 STEP 1876 H=1876:00 +CPU= 18.541
  18:49:16 STEP 1877 H=1877:00 +CPU= 17.795
  18:49:44 STEP 1878 H=1878:00 +CPU= 28.032
  18:50:02 STEP 1879 H=1879:00 +CPU= 17.792
  18:50:21 STEP 1880 H=1880:00 +CPU= 18.328
  18:50:48 STEP 1881 H=1881:00 +CPU= 27.370
  18:51:07 STEP 1882 H=1882:00 +CPU= 18.315
  18:51:25 STEP 1883 H=1883:00 +CPU= 17.814
  18:51:55 STEP 1884 H=1884:00 +CPU= 30.110
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001884
  18:52:13 STEP 1885 H=1885:00 +CPU= 17.858
  18:52:32 STEP 1886 H=1886:00 +CPU= 18.332
  18:52:59 STEP 1887 H=1887:00 +CPU= 27.322
  18:53:18 STEP 1888 H=1888:00 +CPU= 18.317
  18:53:36 STEP 1889 H=1889:00 +CPU= 17.774
  18:54:04 STEP 1890 H=1890:00 +CPU= 27.884
  18:54:22 STEP 1891 H=1891:00 +CPU= 17.769
  18:54:40 STEP 1892 H=1892:00 +CPU= 18.260
  18:55:07 STEP 1893 H=1893:00 +CPU= 27.247
  18:55:26 STEP 1894 H=1894:00 +CPU= 18.272
  18:55:44 STEP 1895 H=1895:00 +CPU= 18.351
  18:56:29 STEP 1896 H=1896:00 +CPU= 32.502
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_8.zip
Uploading the intermediate file: upload_file_8.zip
  18:56:47 STEP 1897 H=1897:00 +CPU= 17.712
Uploading trickle at timestep: 6825600
  18:57:05 STEP 1898 H=1898:00 +CPU= 17.794
  18:57:33 STEP 1899 H=1899:00 +CPU= 27.870
  18:57:52 STEP 1900 H=1900:00 +CPU= 18.320
  18:58:09 STEP 1901 H=1901:00 +CPU= 17.335
  18:58:38 STEP 1902 H=1902:00 +CPU= 28.381
  18:58:56 STEP 1903 H=1903:00 +CPU= 17.913
  18:59:14 STEP 1904 H=1904:00 +CPU= 17.938
  18:59:42 STEP 1905 H=1905:00 +CPU= 27.917
  19:00:01 STEP 1906 H=1906:00 +CPU= 18.500
  19:00:18 STEP 1907 H=1907:00 +CPU= 17.333
  19:00:49 STEP 1908 H=1908:00 +CPU= 30.771
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001908
  19:01:07 STEP 1909 H=1909:00 +CPU= 17.733
  19:01:25 STEP 1910 H=1910:00 +CPU= 17.978
  19:01:54 STEP 1911 H=1911:00 +CPU= 27.871
  19:02:12 STEP 1912 H=1912:00 +CPU= 18.573
  19:02:30 STEP 1913 H=1913:00 +CPU= 17.485
  19:02:59 STEP 1914 H=1914:00 +CPU= 28.299
  19:03:17 STEP 1915 H=1915:00 +CPU= 17.997
  19:03:35 STEP 1916 H=1916:00 +CPU= 17.994
  19:04:03 STEP 1917 H=1917:00 +CPU= 27.914
  19:04:22 STEP 1918 H=1918:00 +CPU= 18.596
  19:04:40 STEP 1919 H=1919:00 +CPU= 17.984
  19:05:13 STEP 1920 H=1920:00 +CPU= 32.294
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001920
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_9.zip
Uploading the intermediate file: upload_file_9.zip
  19:05:31 STEP 1921 H=1921:00 +CPU= 17.805
Uploading trickle at timestep: 6912000
  19:05:49 STEP 1922 H=1922:00 +CPU= 17.738
  19:06:17 STEP 1923 H=1923:00 +CPU= 27.863
  19:06:36 STEP 1924 H=1924:00 +CPU= 18.440
  19:06:53 STEP 1925 H=1925:00 +CPU= 17.382
  19:07:22 STEP 1926 H=1926:00 +CPU= 28.391
  19:07:40 STEP 1927 H=1927:00 +CPU= 17.789
  19:07:58 STEP 1928 H=1928:00 +CPU= 17.897
  19:08:26 STEP 1929 H=1929:00 +CPU= 27.870
  19:08:44 STEP 1930 H=1930:00 +CPU= 18.347
  19:09:02 STEP 1931 H=1931:00 +CPU= 17.403
  19:09:33 STEP 1932 H=1932:00 +CPU= 30.696
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001932
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001932
  19:09:51 STEP 1933 H=1933:00 +CPU= 17.736
  19:10:09 STEP 1934 H=1934:00 +CPU= 17.959
  19:10:37 STEP 1935 H=1935:00 +CPU= 27.766
  19:10:56 STEP 1936 H=1936:00 +CPU= 18.343
  19:11:13 STEP 1937 H=1937:00 +CPU= 17.367
  19:11:42 STEP 1938 H=1938:00 +CPU= 28.399
  19:12:00 STEP 1939 H=1939:00 +CPU= 17.754
  19:12:18 STEP 1940 H=1940:00 +CPU= 17.817
  19:12:46 STEP 1941 H=1941:00 +CPU= 27.764
  19:13:04 STEP 1942 H=1942:00 +CPU= 18.297
  19:13:22 STEP 1943 H=1943:00 +CPU= 17.778
  19:14:00 STEP 1944 H=1944:00 +CPU= 32.674
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_10.zip
Uploading the intermediate file: upload_file_10.zip
  19:14:18 STEP 1945 H=1945:00 +CPU= 17.676
Uploading trickle at timestep: 6998400
  19:14:36 STEP 1946 H=1946:00 +CPU= 17.591
  19:15:05 STEP 1947 H=1947:00 +CPU= 28.033
  19:15:23 STEP 1948 H=1948:00 +CPU= 18.292
  19:15:40 STEP 1949 H=1949:00 +CPU= 17.061
  19:16:09 STEP 1950 H=1950:00 +CPU= 28.603
  19:16:27 STEP 1951 H=1951:00 +CPU= 17.795
  19:16:45 STEP 1952 H=1952:00 +CPU= 17.596
  19:17:13 STEP 1953 H=1953:00 +CPU= 28.024
  19:17:32 STEP 1954 H=1954:00 +CPU= 18.261
  19:17:49 STEP 1955 H=1955:00 +CPU= 17.247
  19:18:20 STEP 1956 H=1956:00 +CPU= 30.815
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001956
  19:18:38 STEP 1957 H=1957:00 +CPU= 17.769
  19:18:56 STEP 1958 H=1958:00 +CPU= 17.599
  19:19:24 STEP 1959 H=1959:00 +CPU= 28.042
  19:19:43 STEP 1960 H=1960:00 +CPU= 18.261
  19:20:00 STEP 1961 H=1961:00 +CPU= 17.075
  19:20:29 STEP 1962 H=1962:00 +CPU= 28.571
  19:20:47 STEP 1963 H=1963:00 +CPU= 17.742
  19:21:04 STEP 1964 H=1964:00 +CPU= 17.571
  19:21:32 STEP 1965 H=1965:00 +CPU= 28.017
  19:21:51 STEP 1966 H=1966:00 +CPU= 18.192
  19:22:09 STEP 1967 H=1967:00 +CPU= 17.751
  19:22:43 STEP 1968 H=1968:00 +CPU= 32.810
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001968
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_11.zip
Uploading the intermediate file: upload_file_11.zip
  19:23:01 STEP 1969 H=1969:00 +CPU= 17.149
Uploading trickle at timestep: 7084800
  19:23:19 STEP 1970 H=1970:00 +CPU= 17.733
  19:23:47 STEP 1971 H=1971:00 +CPU= 27.884
  19:24:06 STEP 1972 H=1972:00 +CPU= 18.249
  19:24:23 STEP 1973 H=1973:00 +CPU= 17.182
  19:24:52 STEP 1974 H=1974:00 +CPU= 28.437
  19:25:09 STEP 1975 H=1975:00 +CPU= 17.670
  19:25:27 STEP 1976 H=1976:00 +CPU= 17.728
  19:25:55 STEP 1977 H=1977:00 +CPU= 27.909
  19:26:14 STEP 1978 H=1978:00 +CPU= 18.243
  19:26:31 STEP 1979 H=1979:00 +CPU= 17.491
  19:27:03 STEP 1980 H=1980:00 +CPU= 30.806
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001980
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001980
  19:27:20 STEP 1981 H=1981:00 +CPU= 17.606
  19:27:38 STEP 1982 H=1982:00 +CPU= 17.752
  19:28:06 STEP 1983 H=1983:00 +CPU= 27.862
  19:28:25 STEP 1984 H=1984:00 +CPU= 18.375
  19:28:42 STEP 1985 H=1985:00 +CPU= 17.214
  19:29:11 STEP 1986 H=1986:00 +CPU= 28.533
  19:29:29 STEP 1987 H=1987:00 +CPU= 17.813
  19:29:47 STEP 1988 H=1988:00 +CPU= 17.702
  19:30:15 STEP 1989 H=1989:00 +CPU= 28.000
  19:30:34 STEP 1990 H=1990:00 +CPU= 18.204
  19:30:52 STEP 1991 H=1991:00 +CPU= 17.853
  19:31:25 STEP 1992 H=1992:00 +CPU= 32.650
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+001992
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_12.zip
Uploading the intermediate file: upload_file_12.zip
  19:31:42 STEP 1993 H=1993:00 +CPU= 17.864
Uploading trickle at timestep: 7171200
  19:32:00 STEP 1994 H=1994:00 +CPU= 17.697
  19:32:29 STEP 1995 H=1995:00 +CPU= 28.045
  19:32:47 STEP 1996 H=1996:00 +CPU= 18.311
  19:33:04 STEP 1997 H=1997:00 +CPU= 17.207
  19:33:33 STEP 1998 H=1998:00 +CPU= 28.565
  19:33:51 STEP 1999 H=1999:00 +CPU= 17.820
  19:34:09 STEP 2000 H=2000:00 +CPU= 17.666
  19:34:37 STEP 2001 H=2001:00 +CPU= 28.019
  19:34:56 STEP 2002 H=2002:00 +CPU= 18.211
  19:35:13 STEP 2003 H=2003:00 +CPU= 17.262
  19:35:44 STEP 2004 H=2004:00 +CPU= 30.921
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002004
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002004
  19:36:02 STEP 2005 H=2005:00 +CPU= 17.869
  19:36:20 STEP 2006 H=2006:00 +CPU= 17.703
  19:36:48 STEP 2007 H=2007:00 +CPU= 28.163
  19:37:07 STEP 2008 H=2008:00 +CPU= 18.445
  19:37:25 STEP 2009 H=2009:00 +CPU= 17.350
  19:37:57 STEP 2010 H=2010:00 +CPU= 31.623
  19:38:18 STEP 2011 H=2011:00 +CPU= 20.170
  19:39:08 STEP 2012 H=2012:00 +CPU= 19.599
  19:39:42 STEP 2013 H=2013:00 +CPU= 31.430
  19:40:03 STEP 2014 H=2014:00 +CPU= 20.906
  19:40:23 STEP 2015 H=2015:00 +CPU= 19.394
  19:41:09 STEP 2016 H=2016:00 +CPU= 36.340
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_13.zip
Uploading the intermediate file: upload_file_13.zip
  19:41:32 STEP 2017 H=2017:00 +CPU= 19.704
Uploading trickle at timestep: 7257600
  19:41:53 STEP 2018 H=2018:00 +CPU= 20.355
  19:42:24 STEP 2019 H=2019:00 +CPU= 31.252
  19:42:45 STEP 2020 H=2020:00 +CPU= 20.543
  19:43:05 STEP 2021 H=2021:00 +CPU= 19.960
  19:43:38 STEP 2022 H=2022:00 +CPU= 31.437
  19:43:58 STEP 2023 H=2023:00 +CPU= 20.062
  19:44:19 STEP 2024 H=2024:00 +CPU= 20.843
  19:44:52 STEP 2025 H=2025:00 +CPU= 31.299
  19:45:13 STEP 2026 H=2026:00 +CPU= 20.684
  19:45:33 STEP 2027 H=2027:00 +CPU= 20.159
  19:46:15 STEP 2028 H=2028:00 +CPU= 33.807
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002028
  19:46:35 STEP 2029 H=2029:00 +CPU= 19.977
  19:46:56 STEP 2030 H=2030:00 +CPU= 20.373
  19:47:27 STEP 2031 H=2031:00 +CPU= 31.293
  19:47:48 STEP 2032 H=2032:00 +CPU= 20.528
  19:48:09 STEP 2033 H=2033:00 +CPU= 20.158
  19:48:51 STEP 2034 H=2034:00 +CPU= 31.510
  19:49:12 STEP 2035 H=2035:00 +CPU= 20.067
  19:49:32 STEP 2036 H=2036:00 +CPU= 20.571
  19:50:14 STEP 2037 H=2037:00 +CPU= 30.865
  19:50:35 STEP 2038 H=2038:00 +CPU= 20.294
  19:50:56 STEP 2039 H=2039:00 +CPU= 20.615
  19:51:39 STEP 2040 H=2040:00 +CPU= 35.497
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMGGhpi1+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMSHhpi1+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12165711/ICMUAhpi1+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_2622_2021050100_123_947_12165711_0_r1142209110_14.zip
Uploading the intermediate file: upload_file_14.zip
  19:51:59 STEP 2041 H=2041:00 +CPU= 19.364
Uploading trickle at timestep: 7344000
  19:52:19 STEP 2042 H=2042:00 +CPU= 19.999
  19:52:51 STEP 2043 H=2043:00 +CPU= 31.263
  19:53:13 STEP 2044 H=2044:00 +CPU= 19.963
  19:53:33 STEP 2045 H=2045:00 +CPU= 19.426
  19:54:05 STEP 2046 H=2046:00 +CPU= 31.932
  19:54:25 STEP 2047 H=2047:00 +CPU= 19.512
  19:54:45 STEP 2048 H=2048:00 +CPU= 20.160
  19:55:17 STEP 2049 H=2049:00 +CPU= 31.410
  19:55:38 STEP 2050 H=2050:00 +CPU= 20.118
  19:55:59 STEP 2051 H=2051:00 +CPU= 19.144
  19:56:34 STEP 2052 H=2052:00 +CPU= 34.344
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhpi1+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhpi1+002052
  19:56:55 STEP 2053 H=2053:00 +CPU= 19.894
  19:57:15 STEP 2054 H=2054:00 +CPU= 20.317
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:09:05 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 7,344,000 7,366 0.0010
09 May 2023 15:09:05 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 7,257,600 6,791 0.0009
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 7,171,200 6,254 0.0009
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 7,084,800 5,738 0.0008
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,998,400 5,217 0.0007
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,912,000 4,702 0.0007
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,825,600 4,184 0.0006
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,739,200 3,668 0.0005
09 May 2023 15:08:34 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,652,800 3,143 0.0005
09 May 2023 15:08:03 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,566,400 2,628 0.0004
09 May 2023 15:08:03 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,480,000 2,104 0.0003
09 May 2023 15:08:03 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,393,600 1,586 0.0002
09 May 2023 15:08:03 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,307,200 1,069 0.0002
09 May 2023 15:08:03 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,220,800 550 0.0001
09 May 2023 15:07:27 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,134,400 33,428 0.0054
09 May 2023 15:07:27 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 6,048,000 32,952 0.0054
09 May 2023 15:07:27 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 5,961,600 32,485 0.0054
09 May 2023 15:07:27 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 5,875,200 32,009 0.0054
09 May 2023 15:07:27 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 5,788,800 31,540 0.0054
09 May 2023 15:07:04 1536641 22247736 oifs_43r3_ps_2622_2021050100_123_947_12165711_0 5,702,400 31,071 0.0054


©2024 climateprediction.net