climateprediction.net home page
Task 22253754

Task 22253754

Name oifs_43r3_ps_0818_1983050100_123_952_12169462_0
Workunit 12169462
Created 21 Dec 2022, 1:46:01 UTC
Sent 9 Jan 2023, 0:46:13 UTC
Report deadline 8 Feb 2023, 0:46:13 UTC
Received 17 Jan 2023, 21:56:18 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1536590
Run time 13 hours 7 min 47 sec
CPU time 12 hours 5 min 53 sec
Validate state Invalid
Credit 9,676.80
Device peak FLOPS 6.89 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,404.57 MB
Peak swap size 4,978.05 MB
Peak disk usage 1,519.13 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
nt/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001536
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001536
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001548
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001548
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001548
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_64.zip
Uploading the intermediate file: upload_file_64.zip
  21:01:45 STEP 1561 H=1561:00 +CPU= 17.193
  21:02:04 STEP 1562 H=1562:00 +CPU= 17.548
  21:02:33 STEP 1563 H=1563:00 +CPU= 26.832
  21:02:52 STEP 1564 H=1564:00 +CPU= 17.590
  21:03:11 STEP 1565 H=1565:00 +CPU= 17.289
  21:03:40 STEP 1566 H=1566:00 +CPU= 27.296
  21:03:59 STEP 1567 H=1567:00 +CPU= 17.227
  21:04:18 STEP 1568 H=1568:00 +CPU= 17.645
  21:04:47 STEP 1569 H=1569:00 +CPU= 26.761
  21:05:06 STEP 1570 H=1570:00 +CPU= 17.648
  21:05:24 STEP 1571 H=1571:00 +CPU= 17.247
  21:05:57 STEP 1572 H=1572:00 +CPU= 29.725
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001572
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001572
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001572
  21:06:15 STEP 1573 H=1573:00 +CPU= 17.297
  21:06:34 STEP 1574 H=1574:00 +CPU= 17.574
  21:07:03 STEP 1575 H=1575:00 +CPU= 26.701
  21:07:22 STEP 1576 H=1576:00 +CPU= 17.608
  21:07:41 STEP 1577 H=1577:00 +CPU= 17.289
  21:08:10 STEP 1578 H=1578:00 +CPU= 27.328
  21:08:29 STEP 1579 H=1579:00 +CPU= 17.115
  21:08:48 STEP 1580 H=1580:00 +CPU= 17.609
  21:09:16 STEP 1581 H=1581:00 +CPU= 26.646
  21:09:35 STEP 1582 H=1582:00 +CPU= 17.549
  21:09:55 STEP 1583 H=1583:00 +CPU= 17.731
  21:10:29 STEP 1584 H=1584:00 +CPU= 31.643
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001584
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001584
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001584
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001560
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001560
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001560
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001572
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001572
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001572
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_65.zip
Uploading the intermediate file: upload_file_65.zip
  21:10:47 STEP 1585 H=1585:00 +CPU= 17.159
  21:11:06 STEP 1586 H=1586:00 +CPU= 17.516
  21:11:35 STEP 1587 H=1587:00 +CPU= 26.745
  21:11:54 STEP 1588 H=1588:00 +CPU= 17.606
  21:12:13 STEP 1589 H=1589:00 +CPU= 17.236
  21:12:42 STEP 1590 H=1590:00 +CPU= 27.266
  21:13:00 STEP 1591 H=1591:00 +CPU= 17.292
  21:13:20 STEP 1592 H=1592:00 +CPU= 17.696
  21:13:48 STEP 1593 H=1593:00 +CPU= 26.674
  21:14:07 STEP 1594 H=1594:00 +CPU= 17.578
  21:14:26 STEP 1595 H=1595:00 +CPU= 17.258
  21:14:58 STEP 1596 H=1596:00 +CPU= 29.707
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001596
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001596
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001596
  21:15:17 STEP 1597 H=1597:00 +CPU= 17.241
  21:15:36 STEP 1598 H=1598:00 +CPU= 17.703
  21:16:05 STEP 1599 H=1599:00 +CPU= 26.757
  21:16:25 STEP 1600 H=1600:00 +CPU= 17.707
  21:16:43 STEP 1601 H=1601:00 +CPU= 17.238
  21:17:13 STEP 1602 H=1602:00 +CPU= 27.142
  21:17:32 STEP 1603 H=1603:00 +CPU= 17.147
  21:17:51 STEP 1604 H=1604:00 +CPU= 17.716
  21:18:20 STEP 1605 H=1605:00 +CPU= 26.765
  21:18:40 STEP 1606 H=1606:00 +CPU= 17.690
  21:18:59 STEP 1607 H=1607:00 +CPU= 17.682
  21:19:33 STEP 1608 H=1608:00 +CPU= 31.528
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001608
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001608
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001608
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001584
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001584
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001584
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001596
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001596
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001596
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_66.zip
Uploading the intermediate file: upload_file_66.zip
  21:19:52 STEP 1609 H=1609:00 +CPU= 17.215
  21:20:11 STEP 1610 H=1610:00 +CPU= 17.646
  21:20:40 STEP 1611 H=1611:00 +CPU= 26.768
  21:21:00 STEP 1612 H=1612:00 +CPU= 17.678
  21:21:19 STEP 1613 H=1613:00 +CPU= 17.283
  21:21:48 STEP 1614 H=1614:00 +CPU= 27.175
  21:22:07 STEP 1615 H=1615:00 +CPU= 17.300
  21:22:27 STEP 1616 H=1616:00 +CPU= 17.703
  21:22:56 STEP 1617 H=1617:00 +CPU= 26.687
  21:23:15 STEP 1618 H=1618:00 +CPU= 17.673
  21:23:34 STEP 1619 H=1619:00 +CPU= 17.265
  21:24:06 STEP 1620 H=1620:00 +CPU= 29.707
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001620
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001620
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001620
  21:24:25 STEP 1621 H=1621:00 +CPU= 17.275
  21:24:44 STEP 1622 H=1622:00 +CPU= 17.687
  21:25:13 STEP 1623 H=1623:00 +CPU= 26.728
  21:25:33 STEP 1624 H=1624:00 +CPU= 17.674
  21:25:51 STEP 1625 H=1625:00 +CPU= 17.187
  21:26:21 STEP 1626 H=1626:00 +CPU= 27.098
  21:26:40 STEP 1627 H=1627:00 +CPU= 17.186
  21:26:59 STEP 1628 H=1628:00 +CPU= 17.669
  21:27:28 STEP 1629 H=1629:00 +CPU= 26.742
  21:27:47 STEP 1630 H=1630:00 +CPU= 17.676
  21:28:07 STEP 1631 H=1631:00 +CPU= 17.748
  21:28:41 STEP 1632 H=1632:00 +CPU= 31.549
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001632
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001632
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001632
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001608
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001608
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001608
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001620
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001620
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001620
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_67.zip
Uploading the intermediate file: upload_file_67.zip
  21:29:00 STEP 1633 H=1633:00 +CPU= 17.210
  21:29:19 STEP 1634 H=1634:00 +CPU= 17.612
  21:29:49 STEP 1635 H=1635:00 +CPU= 26.633
  21:30:08 STEP 1636 H=1636:00 +CPU= 17.612
  21:30:27 STEP 1637 H=1637:00 +CPU= 17.230
  21:30:57 STEP 1638 H=1638:00 +CPU= 27.264
  21:31:16 STEP 1639 H=1639:00 +CPU= 17.239
  21:31:35 STEP 1640 H=1640:00 +CPU= 17.762
  21:32:04 STEP 1641 H=1641:00 +CPU= 26.802
  21:32:24 STEP 1642 H=1642:00 +CPU= 17.697
  21:32:43 STEP 1643 H=1643:00 +CPU= 17.238
  21:33:15 STEP 1644 H=1644:00 +CPU= 29.775
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001644
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001644
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001644
  21:33:34 STEP 1645 H=1645:00 +CPU= 17.222
  21:33:53 STEP 1646 H=1646:00 +CPU= 17.643
  21:34:22 STEP 1647 H=1647:00 +CPU= 26.767
  21:34:41 STEP 1648 H=1648:00 +CPU= 17.752
  21:35:00 STEP 1649 H=1649:00 +CPU= 17.223
  21:35:30 STEP 1650 H=1650:00 +CPU= 27.166
  21:35:48 STEP 1651 H=1651:00 +CPU= 17.193
  21:36:08 STEP 1652 H=1652:00 +CPU= 17.636
  21:36:37 STEP 1653 H=1653:00 +CPU= 26.639
  21:36:56 STEP 1654 H=1654:00 +CPU= 17.695
  21:37:15 STEP 1655 H=1655:00 +CPU= 17.661
  21:37:50 STEP 1656 H=1656:00 +CPU= 31.567
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001656
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001656
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001656
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001632
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001632
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001632
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001644
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001644
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001644
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_68.zip
Uploading the intermediate file: upload_file_68.zip
  21:38:08 STEP 1657 H=1657:00 +CPU= 17.241
  21:38:28 STEP 1658 H=1658:00 +CPU= 17.654
  21:38:57 STEP 1659 H=1659:00 +CPU= 26.638
  21:39:16 STEP 1660 H=1660:00 +CPU= 17.537
  21:39:34 STEP 1661 H=1661:00 +CPU= 17.087
  21:40:04 STEP 1662 H=1662:00 +CPU= 27.200
  21:40:23 STEP 1663 H=1663:00 +CPU= 17.318
  21:40:43 STEP 1664 H=1664:00 +CPU= 17.762
  21:41:12 STEP 1665 H=1665:00 +CPU= 26.702
  21:41:31 STEP 1666 H=1666:00 +CPU= 17.724
  21:41:50 STEP 1667 H=1667:00 +CPU= 17.254
  21:42:22 STEP 1668 H=1668:00 +CPU= 29.710
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001668
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001668
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001668
  21:42:42 STEP 1669 H=1669:00 +CPU= 17.366
  21:43:01 STEP 1670 H=1670:00 +CPU= 17.831
  21:43:31 STEP 1671 H=1671:00 +CPU= 26.886
  21:43:50 STEP 1672 H=1672:00 +CPU= 17.690
  21:44:10 STEP 1673 H=1673:00 +CPU= 17.454
  21:44:39 STEP 1674 H=1674:00 +CPU= 27.214
  21:44:58 STEP 1675 H=1675:00 +CPU= 17.249
  21:45:17 STEP 1676 H=1676:00 +CPU= 17.557
  21:45:46 STEP 1677 H=1677:00 +CPU= 26.792
  21:46:06 STEP 1678 H=1678:00 +CPU= 17.728
  21:46:25 STEP 1679 H=1679:00 +CPU= 17.729
  21:46:59 STEP 1680 H=1680:00 +CPU= 31.620
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001680
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001680
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001680
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001656
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001656
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001656
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001668
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001668
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001668
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_69.zip
Uploading the intermediate file: upload_file_69.zip
  21:47:18 STEP 1681 H=1681:00 +CPU= 17.238
Uploading trickle at timestep: 6048000
  21:47:37 STEP 1682 H=1682:00 +CPU= 17.626
  21:48:07 STEP 1683 H=1683:00 +CPU= 26.991
  21:48:26 STEP 1684 H=1684:00 +CPU= 17.684
  21:48:45 STEP 1685 H=1685:00 +CPU= 17.202
  21:49:14 STEP 1686 H=1686:00 +CPU= 27.167
  21:49:33 STEP 1687 H=1687:00 +CPU= 17.255
  21:49:52 STEP 1688 H=1688:00 +CPU= 17.652
  21:50:21 STEP 1689 H=1689:00 +CPU= 26.772
  21:50:39 STEP 1690 H=1690:00 +CPU= 17.623
  21:50:58 STEP 1691 H=1691:00 +CPU= 17.196
  21:51:30 STEP 1692 H=1692:00 +CPU= 29.706
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001692
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001692
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001692
  21:51:48 STEP 1693 H=1693:00 +CPU= 17.248
  21:52:07 STEP 1694 H=1694:00 +CPU= 17.637
  21:52:36 STEP 1695 H=1695:00 +CPU= 26.731
  21:52:55 STEP 1696 H=1696:00 +CPU= 17.653
  21:53:14 STEP 1697 H=1697:00 +CPU= 17.197
  21:53:43 STEP 1698 H=1698:00 +CPU= 27.209
  21:54:02 STEP 1699 H=1699:00 +CPU= 17.129
  21:54:21 STEP 1700 H=1700:00 +CPU= 17.667
  21:54:49 STEP 1701 H=1701:00 +CPU= 26.699
  21:55:08 STEP 1702 H=1702:00 +CPU= 17.608
  21:55:27 STEP 1703 H=1703:00 +CPU= 17.648
  21:56:02 STEP 1704 H=1704:00 +CPU= 31.700
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001704
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001704
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001704
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001680
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001680
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001680
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001692
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001692
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001692
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_70.zip
Uploading the intermediate file: upload_file_70.zip
  21:56:20 STEP 1705 H=1705:00 +CPU= 17.219
  21:56:39 STEP 1706 H=1706:00 +CPU= 17.636
  21:57:08 STEP 1707 H=1707:00 +CPU= 26.712
  21:57:27 STEP 1708 H=1708:00 +CPU= 17.579
  21:57:45 STEP 1709 H=1709:00 +CPU= 17.194
  21:58:15 STEP 1710 H=1710:00 +CPU= 27.257
  21:58:33 STEP 1711 H=1711:00 +CPU= 17.229
  21:58:53 STEP 1712 H=1712:00 +CPU= 17.750
  21:59:21 STEP 1713 H=1713:00 +CPU= 26.770
  21:59:40 STEP 1714 H=1714:00 +CPU= 17.664
  21:59:59 STEP 1715 H=1715:00 +CPU= 17.274
  22:00:31 STEP 1716 H=1716:00 +CPU= 29.676
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001716
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001716
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001716
  22:00:50 STEP 1717 H=1717:00 +CPU= 17.217
  22:01:09 STEP 1718 H=1718:00 +CPU= 17.641
  22:01:37 STEP 1719 H=1719:00 +CPU= 26.777
  22:01:56 STEP 1720 H=1720:00 +CPU= 17.606
  22:02:15 STEP 1721 H=1721:00 +CPU= 17.162
  22:02:44 STEP 1722 H=1722:00 +CPU= 27.253
  22:03:03 STEP 1723 H=1723:00 +CPU= 17.107
  22:03:22 STEP 1724 H=1724:00 +CPU= 17.614
  22:03:50 STEP 1725 H=1725:00 +CPU= 26.701
  22:04:09 STEP 1726 H=1726:00 +CPU= 17.602
  22:04:28 STEP 1727 H=1727:00 +CPU= 17.656
  22:05:02 STEP 1728 H=1728:00 +CPU= 31.527
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001728
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001728
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001728
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001704
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001704
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001704
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001716
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001716
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001716
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_71.zip
Uploading the intermediate file: upload_file_71.zip
  22:05:21 STEP 1729 H=1729:00 +CPU= 17.239
  22:05:40 STEP 1730 H=1730:00 +CPU= 17.594
  22:06:09 STEP 1731 H=1731:00 +CPU= 26.717
  22:06:27 STEP 1732 H=1732:00 +CPU= 17.550
  22:06:46 STEP 1733 H=1733:00 +CPU= 17.230
  22:07:15 STEP 1734 H=1734:00 +CPU= 27.180
  22:07:34 STEP 1735 H=1735:00 +CPU= 17.187
  22:07:53 STEP 1736 H=1736:00 +CPU= 17.608
  22:08:22 STEP 1737 H=1737:00 +CPU= 26.858
  22:08:41 STEP 1738 H=1738:00 +CPU= 17.692
  22:08:59 STEP 1739 H=1739:00 +CPU= 17.303
  22:09:32 STEP 1740 H=1740:00 +CPU= 29.736
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001740
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001740
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001740
  22:09:50 STEP 1741 H=1741:00 +CPU= 17.267
  22:10:09 STEP 1742 H=1742:00 +CPU= 17.673
  22:10:38 STEP 1743 H=1743:00 +CPU= 26.665
  22:10:57 STEP 1744 H=1744:00 +CPU= 17.622
  22:11:15 STEP 1745 H=1745:00 +CPU= 17.229
  22:11:45 STEP 1746 H=1746:00 +CPU= 27.159
  22:12:03 STEP 1747 H=1747:00 +CPU= 17.199
  22:12:23 STEP 1748 H=1748:00 +CPU= 17.839
  22:12:53 STEP 1749 H=1749:00 +CPU= 26.937
  22:13:12 STEP 1750 H=1750:00 +CPU= 17.667
  22:13:31 STEP 1751 H=1751:00 +CPU= 17.716
  22:14:05 STEP 1752 H=1752:00 +CPU= 31.484
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001752
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001752
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001752
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001728
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001728
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001728
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001740
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001740
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001740
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_72.zip
Uploading the intermediate file: upload_file_72.zip
  22:14:24 STEP 1753 H=1753:00 +CPU= 17.109
  22:14:42 STEP 1754 H=1754:00 +CPU= 17.540
  22:15:11 STEP 1755 H=1755:00 +CPU= 26.677
  22:15:30 STEP 1756 H=1756:00 +CPU= 17.621
  22:15:49 STEP 1757 H=1757:00 +CPU= 17.198
  22:16:18 STEP 1758 H=1758:00 +CPU= 27.186
  22:16:36 STEP 1759 H=1759:00 +CPU= 17.073
  22:16:55 STEP 1760 H=1760:00 +CPU= 17.512
  22:17:24 STEP 1761 H=1761:00 +CPU= 26.720
  22:17:43 STEP 1762 H=1762:00 +CPU= 17.639
  22:18:02 STEP 1763 H=1763:00 +CPU= 17.258
  22:18:34 STEP 1764 H=1764:00 +CPU= 29.829
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001764
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001764
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001764
  22:18:52 STEP 1765 H=1765:00 +CPU= 17.200
  22:19:12 STEP 1766 H=1766:00 +CPU= 17.670
  22:19:40 STEP 1767 H=1767:00 +CPU= 26.758
  22:19:59 STEP 1768 H=1768:00 +CPU= 17.708
  22:20:18 STEP 1769 H=1769:00 +CPU= 17.204
  22:20:47 STEP 1770 H=1770:00 +CPU= 27.144
  22:21:06 STEP 1771 H=1771:00 +CPU= 17.148
  22:21:25 STEP 1772 H=1772:00 +CPU= 17.658
  22:21:53 STEP 1773 H=1773:00 +CPU= 26.801
  22:22:12 STEP 1774 H=1774:00 +CPU= 17.557
  22:22:31 STEP 1775 H=1775:00 +CPU= 17.764
  22:23:05 STEP 1776 H=1776:00 +CPU= 31.563
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001776
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001776
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001776
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001752
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001752
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001752
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001764
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001764
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001764
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_73.zip
Uploading the intermediate file: upload_file_73.zip
  22:23:24 STEP 1777 H=1777:00 +CPU= 17.201
  22:23:43 STEP 1778 H=1778:00 +CPU= 17.587
  22:24:12 STEP 1779 H=1779:00 +CPU= 26.750
  22:24:31 STEP 1780 H=1780:00 +CPU= 17.611
  22:24:49 STEP 1781 H=1781:00 +CPU= 17.152
  22:25:18 STEP 1782 H=1782:00 +CPU= 27.250
  22:25:37 STEP 1783 H=1783:00 +CPU= 17.227
  22:25:56 STEP 1784 H=1784:00 +CPU= 17.569
  22:26:24 STEP 1785 H=1785:00 +CPU= 26.662
  22:26:43 STEP 1786 H=1786:00 +CPU= 17.667
  22:27:02 STEP 1787 H=1787:00 +CPU= 17.244
  22:27:34 STEP 1788 H=1788:00 +CPU= 29.679
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001788
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001788
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001788
  22:27:53 STEP 1789 H=1789:00 +CPU= 17.179
  22:28:12 STEP 1790 H=1790:00 +CPU= 17.706
  22:28:41 STEP 1791 H=1791:00 +CPU= 26.750
  22:29:00 STEP 1792 H=1792:00 +CPU= 17.527
  22:29:18 STEP 1793 H=1793:00 +CPU= 17.179
  22:29:47 STEP 1794 H=1794:00 +CPU= 27.217
  22:30:06 STEP 1795 H=1795:00 +CPU= 17.206
  22:30:25 STEP 1796 H=1796:00 +CPU= 17.561
  22:30:54 STEP 1797 H=1797:00 +CPU= 26.668
  22:31:12 STEP 1798 H=1798:00 +CPU= 17.492
  22:31:31 STEP 1799 H=1799:00 +CPU= 17.656
  22:32:06 STEP 1800 H=1800:00 +CPU= 31.648
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001800
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001800
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001800
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001776
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001776
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001776
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001788
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001788
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001788
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_74.zip
Uploading the intermediate file: upload_file_74.zip
  22:32:24 STEP 1801 H=1801:00 +CPU= 17.173
  22:32:43 STEP 1802 H=1802:00 +CPU= 17.560
  22:33:12 STEP 1803 H=1803:00 +CPU= 26.677
  22:33:31 STEP 1804 H=1804:00 +CPU= 17.622
  22:33:49 STEP 1805 H=1805:00 +CPU= 17.109
  22:34:19 STEP 1806 H=1806:00 +CPU= 27.269
  22:34:39 STEP 1807 H=1807:00 +CPU= 17.459
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:35:10 STEP 1808 H=1808:00 +CPU= 17.918
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:36:13 STEP 1809 H=1809:00 +CPU= 27.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:36:44 STEP 1810 H=1810:00 +CPU= 17.888
  22:37:03 STEP 1811 H=1811:00 +CPU= 17.434
  22:37:36 STEP 1812 H=1812:00 +CPU= 29.860
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001812
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001812
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001812
  22:37:55 STEP 1813 H=1813:00 +CPU= 17.318
  22:38:14 STEP 1814 H=1814:00 +CPU= 17.734
  22:38:42 STEP 1815 H=1815:00 +CPU= 26.757
  22:39:01 STEP 1816 H=1816:00 +CPU= 17.611
  22:39:20 STEP 1817 H=1817:00 +CPU= 17.223
  22:39:49 STEP 1818 H=1818:00 +CPU= 27.296
  22:40:08 STEP 1819 H=1819:00 +CPU= 17.170
  22:40:27 STEP 1820 H=1820:00 +CPU= 17.625
  22:40:56 STEP 1821 H=1821:00 +CPU= 26.842
  22:41:15 STEP 1822 H=1822:00 +CPU= 17.617
  22:41:34 STEP 1823 H=1823:00 +CPU= 17.839
  22:42:08 STEP 1824 H=1824:00 +CPU= 31.635
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001824
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001824
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001824
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001800
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001800
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001800
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001812
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001812
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001812
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_75.zip
Uploading the intermediate file: upload_file_75.zip
  22:42:27 STEP 1825 H=1825:00 +CPU= 17.180
  22:42:46 STEP 1826 H=1826:00 +CPU= 17.668
  22:43:14 STEP 1827 H=1827:00 +CPU= 26.705
  22:43:34 STEP 1828 H=1828:00 +CPU= 17.707
  22:43:52 STEP 1829 H=1829:00 +CPU= 17.153
  22:44:22 STEP 1830 H=1830:00 +CPU= 27.213
  22:44:40 STEP 1831 H=1831:00 +CPU= 17.182
  22:45:00 STEP 1832 H=1832:00 +CPU= 17.787
  22:45:29 STEP 1833 H=1833:00 +CPU= 26.766
  22:45:48 STEP 1834 H=1834:00 +CPU= 17.616
  22:46:07 STEP 1835 H=1835:00 +CPU= 17.122
  22:46:39 STEP 1836 H=1836:00 +CPU= 29.708
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001836
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001836
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001836
  22:46:58 STEP 1837 H=1837:00 +CPU= 17.204
  22:47:17 STEP 1838 H=1838:00 +CPU= 17.661
  22:47:46 STEP 1839 H=1839:00 +CPU= 26.740
  22:48:05 STEP 1840 H=1840:00 +CPU= 17.642
  22:48:24 STEP 1841 H=1841:00 +CPU= 17.201
  22:48:54 STEP 1842 H=1842:00 +CPU= 27.216
  22:49:13 STEP 1843 H=1843:00 +CPU= 17.283
  22:49:32 STEP 1844 H=1844:00 +CPU= 17.691
  22:50:01 STEP 1845 H=1845:00 +CPU= 26.620
  22:50:20 STEP 1846 H=1846:00 +CPU= 17.590
  22:50:39 STEP 1847 H=1847:00 +CPU= 17.708
  22:51:14 STEP 1848 H=1848:00 +CPU= 31.588
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001848
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001848
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001848
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001824
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001824
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001824
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001836
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001836
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001836
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_76.zip
Uploading the intermediate file: upload_file_76.zip
  22:51:33 STEP 1849 H=1849:00 +CPU= 17.158
  22:51:52 STEP 1850 H=1850:00 +CPU= 17.633
  22:52:21 STEP 1851 H=1851:00 +CPU= 26.760
  22:52:41 STEP 1852 H=1852:00 +CPU= 17.645
  22:52:59 STEP 1853 H=1853:00 +CPU= 17.201
  22:53:29 STEP 1854 H=1854:00 +CPU= 27.313
  22:53:48 STEP 1855 H=1855:00 +CPU= 17.246
  22:54:07 STEP 1856 H=1856:00 +CPU= 17.772
  22:54:36 STEP 1857 H=1857:00 +CPU= 26.741
  22:54:56 STEP 1858 H=1858:00 +CPU= 17.737
  22:55:15 STEP 1859 H=1859:00 +CPU= 17.268
  22:55:47 STEP 1860 H=1860:00 +CPU= 29.661
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001860
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001860
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001860
  22:56:06 STEP 1861 H=1861:00 +CPU= 17.242
  22:56:25 STEP 1862 H=1862:00 +CPU= 17.830
  22:56:55 STEP 1863 H=1863:00 +CPU= 26.802
  22:57:14 STEP 1864 H=1864:00 +CPU= 17.669
  22:57:33 STEP 1865 H=1865:00 +CPU= 17.314
  22:58:02 STEP 1866 H=1866:00 +CPU= 27.276
  22:58:21 STEP 1867 H=1867:00 +CPU= 17.235
  22:58:40 STEP 1868 H=1868:00 +CPU= 17.610
  22:59:09 STEP 1869 H=1869:00 +CPU= 26.694
  22:59:28 STEP 1870 H=1870:00 +CPU= 17.662
  22:59:47 STEP 1871 H=1871:00 +CPU= 17.666
  23:00:21 STEP 1872 H=1872:00 +CPU= 31.503
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001872
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001872
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001872
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001848
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001848
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001848
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001860
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001860
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001860
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_77.zip
Uploading the intermediate file: upload_file_77.zip
  23:00:40 STEP 1873 H=1873:00 +CPU= 17.170
  23:00:59 STEP 1874 H=1874:00 +CPU= 17.583
  23:01:27 STEP 1875 H=1875:00 +CPU= 26.688
  23:01:46 STEP 1876 H=1876:00 +CPU= 17.523
  23:02:04 STEP 1877 H=1877:00 +CPU= 17.207
  23:02:34 STEP 1878 H=1878:00 +CPU= 27.213
  23:02:52 STEP 1879 H=1879:00 +CPU= 17.167
  23:03:11 STEP 1880 H=1880:00 +CPU= 17.732
  23:03:40 STEP 1881 H=1881:00 +CPU= 26.597
  23:03:59 STEP 1882 H=1882:00 +CPU= 17.603
  23:04:17 STEP 1883 H=1883:00 +CPU= 17.169
  23:04:49 STEP 1884 H=1884:00 +CPU= 29.616
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001884
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001884
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001884
  23:05:08 STEP 1885 H=1885:00 +CPU= 17.120
  23:05:27 STEP 1886 H=1886:00 +CPU= 17.566
  23:05:56 STEP 1887 H=1887:00 +CPU= 26.637
  23:06:15 STEP 1888 H=1888:00 +CPU= 17.571
  23:06:34 STEP 1889 H=1889:00 +CPU= 17.184
  23:07:03 STEP 1890 H=1890:00 +CPU= 27.035
  23:07:21 STEP 1891 H=1891:00 +CPU= 17.134
  23:07:41 STEP 1892 H=1892:00 +CPU= 17.542
  23:08:09 STEP 1893 H=1893:00 +CPU= 26.522
  23:08:28 STEP 1894 H=1894:00 +CPU= 17.557
  23:08:47 STEP 1895 H=1895:00 +CPU= 17.544
  23:09:22 STEP 1896 H=1896:00 +CPU= 31.458
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001896
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001896
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001896
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001872
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001872
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001872
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001884
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001884
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001884
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_78.zip
Uploading the intermediate file: upload_file_78.zip
  23:09:40 STEP 1897 H=1897:00 +CPU= 17.169
  23:10:00 STEP 1898 H=1898:00 +CPU= 17.647
  23:10:28 STEP 1899 H=1899:00 +CPU= 26.594
  23:10:47 STEP 1900 H=1900:00 +CPU= 17.548
  23:11:06 STEP 1901 H=1901:00 +CPU= 17.208
  23:11:35 STEP 1902 H=1902:00 +CPU= 27.110
  23:11:54 STEP 1903 H=1903:00 +CPU= 17.093
  23:12:13 STEP 1904 H=1904:00 +CPU= 17.678
  23:12:42 STEP 1905 H=1905:00 +CPU= 26.605
  23:13:01 STEP 1906 H=1906:00 +CPU= 17.525
  23:13:19 STEP 1907 H=1907:00 +CPU= 17.187
  23:13:51 STEP 1908 H=1908:00 +CPU= 29.530
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001908
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001908
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001908
  23:14:10 STEP 1909 H=1909:00 +CPU= 17.123
  23:14:29 STEP 1910 H=1910:00 +CPU= 17.582
  23:14:58 STEP 1911 H=1911:00 +CPU= 26.710
  23:15:17 STEP 1912 H=1912:00 +CPU= 17.619
  23:15:36 STEP 1913 H=1913:00 +CPU= 17.081
  23:16:05 STEP 1914 H=1914:00 +CPU= 26.990
  23:16:24 STEP 1915 H=1915:00 +CPU= 17.124
  23:16:43 STEP 1916 H=1916:00 +CPU= 17.540
  23:17:12 STEP 1917 H=1917:00 +CPU= 26.661
  23:17:31 STEP 1918 H=1918:00 +CPU= 17.530
  23:17:49 STEP 1919 H=1919:00 +CPU= 17.514
  23:18:24 STEP 1920 H=1920:00 +CPU= 31.505
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001920
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001920
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001920
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001896
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001896
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001896
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001908
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001908
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001908
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_79.zip
Uploading the intermediate file: upload_file_79.zip
  23:18:42 STEP 1921 H=1921:00 +CPU= 17.128
Uploading trickle at timestep: 6912000
  23:19:02 STEP 1922 H=1922:00 +CPU= 17.596
  23:19:31 STEP 1923 H=1923:00 +CPU= 26.576
  23:19:50 STEP 1924 H=1924:00 +CPU= 17.521
  23:20:08 STEP 1925 H=1925:00 +CPU= 17.100
  23:20:37 STEP 1926 H=1926:00 +CPU= 27.136
  23:20:56 STEP 1927 H=1927:00 +CPU= 17.166
  23:21:15 STEP 1928 H=1928:00 +CPU= 17.760
  23:21:44 STEP 1929 H=1929:00 +CPU= 26.781
  23:22:04 STEP 1930 H=1930:00 +CPU= 17.609
  23:22:22 STEP 1931 H=1931:00 +CPU= 17.441
  23:22:54 STEP 1932 H=1932:00 +CPU= 29.774
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001932
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001932
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001932
  23:23:13 STEP 1933 H=1933:00 +CPU= 17.118
  23:23:32 STEP 1934 H=1934:00 +CPU= 17.596
  23:24:01 STEP 1935 H=1935:00 +CPU= 26.705
  23:24:20 STEP 1936 H=1936:00 +CPU= 17.642
  23:24:39 STEP 1937 H=1937:00 +CPU= 17.189
  23:25:08 STEP 1938 H=1938:00 +CPU= 27.135
  23:25:27 STEP 1939 H=1939:00 +CPU= 17.224
  23:25:46 STEP 1940 H=1940:00 +CPU= 17.532
  23:26:15 STEP 1941 H=1941:00 +CPU= 26.745
  23:26:34 STEP 1942 H=1942:00 +CPU= 17.629
  23:26:53 STEP 1943 H=1943:00 +CPU= 17.616
  23:27:28 STEP 1944 H=1944:00 +CPU= 31.532
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001944
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001944
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001944
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001920
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001920
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001920
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001932
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001932
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001932
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_80.zip
Uploading the intermediate file: upload_file_80.zip
  23:27:47 STEP 1945 H=1945:00 +CPU= 17.319
  23:28:06 STEP 1946 H=1946:00 +CPU= 17.783
  23:28:35 STEP 1947 H=1947:00 +CPU= 26.799
  23:28:54 STEP 1948 H=1948:00 +CPU= 17.731
  23:29:13 STEP 1949 H=1949:00 +CPU= 17.255
  23:29:43 STEP 1950 H=1950:00 +CPU= 27.298
  23:30:01 STEP 1951 H=1951:00 +CPU= 17.263
  23:30:21 STEP 1952 H=1952:00 +CPU= 17.838
  23:30:50 STEP 1953 H=1953:00 +CPU= 26.753
  23:31:09 STEP 1954 H=1954:00 +CPU= 17.715
  23:31:28 STEP 1955 H=1955:00 +CPU= 17.255
  23:32:00 STEP 1956 H=1956:00 +CPU= 29.688
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001956
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001956
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001956
  23:32:19 STEP 1957 H=1957:00 +CPU= 17.322
  23:32:38 STEP 1958 H=1958:00 +CPU= 17.744
  23:33:07 STEP 1959 H=1959:00 +CPU= 26.947
  23:33:27 STEP 1960 H=1960:00 +CPU= 17.751
  23:33:45 STEP 1961 H=1961:00 +CPU= 17.331
  23:34:15 STEP 1962 H=1962:00 +CPU= 27.343
  23:34:34 STEP 1963 H=1963:00 +CPU= 17.192
  23:34:53 STEP 1964 H=1964:00 +CPU= 17.623
  23:35:22 STEP 1965 H=1965:00 +CPU= 26.668
  23:35:41 STEP 1966 H=1966:00 +CPU= 17.558
  23:36:00 STEP 1967 H=1967:00 +CPU= 17.679
  23:36:34 STEP 1968 H=1968:00 +CPU= 31.464
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001968
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001968
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001968
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001944
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001944
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001944
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001956
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001956
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001956
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_81.zip
Uploading the intermediate file: upload_file_81.zip
  23:36:52 STEP 1969 H=1969:00 +CPU= 17.134
  23:37:12 STEP 1970 H=1970:00 +CPU= 17.644
  23:37:42 STEP 1971 H=1971:00 +CPU= 26.984
  23:38:01 STEP 1972 H=1972:00 +CPU= 17.751
  23:38:20 STEP 1973 H=1973:00 +CPU= 17.361
  23:38:51 STEP 1974 H=1974:00 +CPU= 27.634
  23:39:10 STEP 1975 H=1975:00 +CPU= 17.381
  23:39:29 STEP 1976 H=1976:00 +CPU= 17.873
  23:39:58 STEP 1977 H=1977:00 +CPU= 26.884
  23:40:17 STEP 1978 H=1978:00 +CPU= 17.798
  23:40:36 STEP 1979 H=1979:00 +CPU= 17.525
  23:41:09 STEP 1980 H=1980:00 +CPU= 29.950
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001980
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001980
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001980
  23:41:28 STEP 1981 H=1981:00 +CPU= 17.360
  23:41:47 STEP 1982 H=1982:00 +CPU= 17.968
  23:42:17 STEP 1983 H=1983:00 +CPU= 27.026
  23:42:37 STEP 1984 H=1984:00 +CPU= 17.962
  23:42:56 STEP 1985 H=1985:00 +CPU= 17.477
  23:43:26 STEP 1986 H=1986:00 +CPU= 27.642
  23:43:45 STEP 1987 H=1987:00 +CPU= 17.643
  23:44:04 STEP 1988 H=1988:00 +CPU= 18.221
  23:44:33 STEP 1989 H=1989:00 +CPU= 27.190
  23:44:51 STEP 1990 H=1990:00 +CPU= 17.870
  23:45:10 STEP 1991 H=1991:00 +CPU= 18.148
  23:45:43 STEP 1992 H=1992:00 +CPU= 32.201
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+001992
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+001992
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+001992
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001968
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001968
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001968
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001980
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001980
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001980
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_82.zip
Uploading the intermediate file: upload_file_82.zip
  23:46:01 STEP 1993 H=1993:00 +CPU= 17.439
  23:46:20 STEP 1994 H=1994:00 +CPU= 17.823
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46492.928] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.112] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.112] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.112] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.112] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.112] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.112] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234640:1673959600:46493.114] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234641:1673959601:46494.150] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:JJ:1:1:8419:8419] [20230117:234641:1673959601:46494.150] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000010337EE  cloudsc_                 3024  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F1A790  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1983050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0818
(argv4) batchid: 952
(argv5) wuid: 12169462
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/inst_data/boinc-client/slots/10
Project directory is: /mnt/inst_data/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462
..mkdir for temp folder for results failed
Copying: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/inst_data/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/inst_data/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7673ae9fd0d2fc0bbefee19dda59e49c to: /mnt/inst_data/boinc-client/slots/10/oifs_43r3_ps_0818_1983050100_123_952_12169462.zip
Unzipping the namelist zip file: /mnt/inst_data/boinc-client/slots/10/oifs_43r3_ps_0818_1983050100_123_952_12169462.zip
ic_ancil_file: ic_ancil_12169462
ifsdata_file: ifsdata_12169462
climate_data_file: clim_data_12169462
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b74f4a7d51d715dd22f4575807efd5ca to: /mnt/inst_data/boinc-client/slots/10/ic_ancil_12169462.zip
Unzipping the IC ancils zip file: /mnt/inst_data/boinc-client/slots/10/ic_ancil_12169462.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9dc91f728ae42dd71b99a5eaa540fdbc to: /mnt/inst_data/boinc-client/slots/10/ifsdata/ifsdata_12169462.zip
Unzipping the ifsdata_zip file: /mnt/inst_data/boinc-client/slots/10/ifsdata/ifsdata_12169462.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/inst_data/boinc-client/slots/10/159l_2/clim_data_12169462.zip
Unzipping the climate data zip file: /mnt/inst_data/boinc-client/slots/10/159l_2/clim_data_12169462.zip
Checking for progress XML file: /mnt/inst_data/boinc-client/slots/10/progress_file_12169462.xml
Opened progress file ok : /mnt/inst_data/boinc-client/slots/10/progress_file_12169462.xml
-- Model is restarting --
Adjusting last_iter, 1995, to previous model restart step.
Creating progress file: /mnt/inst_data/boinc-client/slots/10/progress_file_12169462.xml
last_cpu_time: 43010
upload_file_number: 83
last_iter: 1991
last_upload: 7171200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610
The child process has been launched with process id: 3001
Executing the command: /mnt/inst_data/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:JJ:1:1:3001:3001] [20230118:084231:1673991751:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:43:08 STEP 1992 H=1992:00 +CPU= 26.611
  08:43:25 STEP 1993 H=1993:00 +CPU= 16.249
  08:43:42 STEP 1994 H=1994:00 +CPU= 16.919
  08:44:08 STEP 1995 H=1995:00 +CPU= 25.570
  08:44:25 STEP 1996 H=1996:00 +CPU= 16.832
  08:44:42 STEP 1997 H=1997:00 +CPU= 16.485
  08:45:08 STEP 1998 H=1998:00 +CPU= 26.014
  08:45:25 STEP 1999 H=1999:00 +CPU= 16.890
  08:45:43 STEP 2000 H=2000:00 +CPU= 16.918
  08:46:09 STEP 2001 H=2001:00 +CPU= 26.015
  08:46:26 STEP 2002 H=2002:00 +CPU= 16.796
  08:46:43 STEP 2003 H=2003:00 +CPU= 16.539
  08:47:13 STEP 2004 H=2004:00 +CPU= 28.607
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+002004
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+002004
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+002004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:47:42 STEP 2005 H=2005:00 +CPU= 17.150
  08:47:59 STEP 2006 H=2006:00 +CPU= 17.092
  08:48:26 STEP 2007 H=2007:00 +CPU= 26.002
  08:48:44 STEP 2008 H=2008:00 +CPU= 17.226
  08:49:01 STEP 2009 H=2009:00 +CPU= 16.694
  08:49:29 STEP 2010 H=2010:00 +CPU= 26.710
  08:49:46 STEP 2011 H=2011:00 +CPU= 16.578
  08:50:03 STEP 2012 H=2012:00 +CPU= 16.917
  08:50:29 STEP 2013 H=2013:00 +CPU= 25.816
  08:50:46 STEP 2014 H=2014:00 +CPU= 17.042
  08:51:04 STEP 2015 H=2015:00 +CPU= 16.921
  08:51:35 STEP 2016 H=2016:00 +CPU= 30.468
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMGGhq0f+002016
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMSHhq0f+002016
Moving to projects directory: /mnt/inst_data/boinc-client/slots/10/ICMUAhq0f+002016
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+001992
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+001992
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+001992
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMGGhq0f+002004
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMSHhq0f+002004
Adding to the zip: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169462/ICMUAhq0f+002004
Zipping up the intermediate file: /mnt/inst_data/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0818_1983050100_123_952_12169462_0_r254580610_83.zip
Uploading the intermediate file: upload_file_83.zip
  08:51:52 STEP 2017 H=2017:00 +CPU= 16.495
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:24:10 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 6,912,000 41,451 0.0060
09 May 2023 16:24:03 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 6,048,000 36,425 0.0060
09 May 2023 16:23:54 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 5,184,000 31,402 0.0061
09 May 2023 16:23:47 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 4,320,000 26,357 0.0061
09 May 2023 16:23:39 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 3,456,000 21,261 0.0062
09 May 2023 16:23:33 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 2,592,000 16,097 0.0062
09 May 2023 16:23:25 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 1,728,000 10,921 0.0063
09 May 2023 16:23:18 1536590 22253754 oifs_43r3_ps_0818_1983050100_123_952_12169462_0 864,000 5,766 0.0067


©2024 climateprediction.net