climateprediction.net home page
Task 22446291

Task 22446291

Name oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0
Workunit 12292682
Created 13 Jun 2024, 2:19:49 UTC
Sent 13 Jun 2024, 17:58:57 UTC
Report deadline 12 Aug 2024, 17:58:57 UTC
Received 15 Jun 2024, 0:06:19 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537687
Run time 16 hours 45 min 24 sec
CPU time 16 hours 45 min 24 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 7.20 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.66 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,360.12 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:864502:864502] [20240614:203611:1718397371:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:36:44 STEP 1584 H= 396:00 +CPU= 28.444
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001584
  20:37:11 STEP 1585 H= 396:15 +CPU= 26.283
  20:37:37 STEP 1586 H= 396:30 +CPU= 26.365
  20:38:01 STEP 1587 H= 396:45 +CPU= 24.369
  20:38:27 STEP 1588 H= 397:00 +CPU= 25.861
  20:38:54 STEP 1589 H= 397:15 +CPU= 26.555
  20:39:23 STEP 1590 H= 397:30 +CPU= 28.487
  20:39:52 STEP 1591 H= 397:45 +CPU= 28.730
  20:40:22 STEP 1592 H= 398:00 +CPU= 29.226
  20:40:51 STEP 1593 H= 398:15 +CPU= 28.261
  20:41:20 STEP 1594 H= 398:30 +CPU= 28.128
  20:41:48 STEP 1595 H= 398:45 +CPU= 28.339
  20:42:25 STEP 1596 H= 399:00 +CPU= 35.980
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001596
  20:42:54 STEP 1597 H= 399:15 +CPU= 29.084
  20:43:24 STEP 1598 H= 399:30 +CPU= 29.083
  20:43:53 STEP 1599 H= 399:45 +CPU= 28.816
  20:44:23 STEP 1600 H= 400:00 +CPU= 29.184
  20:44:52 STEP 1601 H= 400:15 +CPU= 28.757
  20:45:21 STEP 1602 H= 400:30 +CPU= 28.473
  20:45:50 STEP 1603 H= 400:45 +CPU= 29.155
  20:46:20 STEP 1604 H= 401:00 +CPU= 29.440
  20:46:50 STEP 1605 H= 401:15 +CPU= 28.949
  20:47:19 STEP 1606 H= 401:30 +CPU= 28.521
  20:47:48 STEP 1607 H= 401:45 +CPU= 28.521
  20:48:25 STEP 1608 H= 402:00 +CPU= 36.077
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001608
  20:48:54 STEP 1609 H= 402:15 +CPU= 28.444
  20:49:21 STEP 1610 H= 402:30 +CPU= 27.374
  20:49:50 STEP 1611 H= 402:45 +CPU= 28.543
  20:50:20 STEP 1612 H= 403:00 +CPU= 29.006
  20:50:49 STEP 1613 H= 403:15 +CPU= 28.570
  20:51:18 STEP 1614 H= 403:30 +CPU= 28.713
  20:51:47 STEP 1615 H= 403:45 +CPU= 28.670
  20:52:17 STEP 1616 H= 404:00 +CPU= 29.809
  20:52:46 STEP 1617 H= 404:15 +CPU= 28.892
  20:53:15 STEP 1618 H= 404:30 +CPU= 28.860
  20:53:43 STEP 1619 H= 404:45 +CPU= 27.743
  20:54:11 STEP 1620 H= 405:00 +CPU= 28.552
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001620
  20:54:35 STEP 1621 H= 405:15 +CPU= 23.630
  20:55:00 STEP 1622 H= 405:30 +CPU= 24.476
  20:55:23 STEP 1623 H= 405:45 +CPU= 23.683
  20:55:47 STEP 1624 H= 406:00 +CPU= 24.147
  20:56:13 STEP 1625 H= 406:15 +CPU= 25.545
  20:56:39 STEP 1626 H= 406:30 +CPU= 25.937
  20:57:08 STEP 1627 H= 406:45 +CPU= 28.630
  20:57:38 STEP 1628 H= 407:00 +CPU= 29.643
  20:58:08 STEP 1629 H= 407:15 +CPU= 28.792
  20:58:38 STEP 1630 H= 407:30 +CPU= 28.793
  20:59:05 STEP 1631 H= 407:45 +CPU= 27.219
  20:59:41 STEP 1632 H= 408:00 +CPU= 36.329
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0_r492272722_16.zip
Uploading the intermediate file: upload_file_16.zip
  21:00:08 STEP 1633 H= 408:15 +CPU= 25.734
  21:00:33 STEP 1634 H= 408:30 +CPU= 25.122
  21:00:58 STEP 1635 H= 408:45 +CPU= 25.101
  21:01:23 STEP 1636 H= 409:00 +CPU= 24.941
  21:01:49 STEP 1637 H= 409:15 +CPU= 25.285
  21:02:15 STEP 1638 H= 409:30 +CPU= 26.197
  21:02:44 STEP 1639 H= 409:45 +CPU= 27.638
  21:03:13 STEP 1640 H= 410:00 +CPU= 29.076
  21:03:42 STEP 1641 H= 410:15 +CPU= 28.981
  21:04:11 STEP 1642 H= 410:30 +CPU= 28.630
  21:04:39 STEP 1643 H= 410:45 +CPU= 28.314
  21:05:15 STEP 1644 H= 411:00 +CPU= 35.107
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001644
  21:05:43 STEP 1645 H= 411:15 +CPU= 28.296
  21:06:12 STEP 1646 H= 411:30 +CPU= 28.697
  21:06:40 STEP 1647 H= 411:45 +CPU= 27.881
  21:07:09 STEP 1648 H= 412:00 +CPU= 28.766
  21:07:38 STEP 1649 H= 412:15 +CPU= 28.629
  21:08:07 STEP 1650 H= 412:30 +CPU= 28.357
  21:08:35 STEP 1651 H= 412:45 +CPU= 27.584
  21:09:04 STEP 1652 H= 413:00 +CPU= 29.146
  21:09:25 STEP 1653 H= 413:15 +CPU= 21.422
  21:09:46 STEP 1654 H= 413:30 +CPU= 20.946
  21:10:06 STEP 1655 H= 413:45 +CPU= 19.414
  21:10:36 STEP 1656 H= 414:00 +CPU= 29.791
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001656
  21:11:00 STEP 1657 H= 414:15 +CPU= 24.119
  21:11:26 STEP 1658 H= 414:30 +CPU= 25.971
  21:11:51 STEP 1659 H= 414:45 +CPU= 25.254
  21:12:19 STEP 1660 H= 415:00 +CPU= 27.401
  21:12:48 STEP 1661 H= 415:15 +CPU= 28.168
  21:13:17 STEP 1662 H= 415:30 +CPU= 29.310
  21:13:46 STEP 1663 H= 415:45 +CPU= 28.520
  21:14:15 STEP 1664 H= 416:00 +CPU= 29.321
  21:14:44 STEP 1665 H= 416:15 +CPU= 28.478
  21:15:13 STEP 1666 H= 416:30 +CPU= 28.563
  21:15:40 STEP 1667 H= 416:45 +CPU= 26.334
  21:16:06 STEP 1668 H= 417:00 +CPU= 26.047
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001668
  21:16:27 STEP 1669 H= 417:15 +CPU= 21.318
  21:16:51 STEP 1670 H= 417:30 +CPU= 23.898
  21:17:16 STEP 1671 H= 417:45 +CPU= 25.347
  21:17:41 STEP 1672 H= 418:00 +CPU= 24.989
  21:18:08 STEP 1673 H= 418:15 +CPU= 25.491
  21:18:32 STEP 1674 H= 418:30 +CPU= 24.327
  21:18:59 STEP 1675 H= 418:45 +CPU= 26.628
  21:19:29 STEP 1676 H= 419:00 +CPU= 28.931
  21:19:58 STEP 1677 H= 419:15 +CPU= 28.732
  21:20:27 STEP 1678 H= 419:30 +CPU= 28.701
  21:20:56 STEP 1679 H= 419:45 +CPU= 28.392
  21:21:34 STEP 1680 H= 420:00 +CPU= 38.078
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001680
  21:22:03 STEP 1681 H= 420:15 +CPU= 28.207
  21:22:31 STEP 1682 H= 420:30 +CPU= 28.956
  21:23:01 STEP 1683 H= 420:45 +CPU= 28.921
  21:23:30 STEP 1684 H= 421:00 +CPU= 28.968
  21:23:59 STEP 1685 H= 421:15 +CPU= 28.249
  21:24:28 STEP 1686 H= 421:30 +CPU= 28.738
  21:24:57 STEP 1687 H= 421:45 +CPU= 28.386
  21:25:26 STEP 1688 H= 422:00 +CPU= 29.386
  21:25:55 STEP 1689 H= 422:15 +CPU= 28.908
  21:26:24 STEP 1690 H= 422:30 +CPU= 28.751
  21:26:53 STEP 1691 H= 422:45 +CPU= 28.599
  21:27:30 STEP 1692 H= 423:00 +CPU= 35.889
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001692
  21:27:58 STEP 1693 H= 423:15 +CPU= 28.369
  21:28:27 STEP 1694 H= 423:30 +CPU= 28.917
  21:28:56 STEP 1695 H= 423:45 +CPU= 28.166
  21:29:25 STEP 1696 H= 424:00 +CPU= 29.083
  21:29:54 STEP 1697 H= 424:15 +CPU= 28.577
  21:30:22 STEP 1698 H= 424:30 +CPU= 28.666
  21:30:51 STEP 1699 H= 424:45 +CPU= 28.766
  21:31:21 STEP 1700 H= 425:00 +CPU= 28.999
  21:31:50 STEP 1701 H= 425:15 +CPU= 28.526
  21:32:20 STEP 1702 H= 425:30 +CPU= 28.595
  21:32:48 STEP 1703 H= 425:45 +CPU= 28.615
  21:33:24 STEP 1704 H= 426:00 +CPU= 35.812
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001704
  21:33:54 STEP 1705 H= 426:15 +CPU= 28.852
  21:34:23 STEP 1706 H= 426:30 +CPU= 28.569
  21:34:51 STEP 1707 H= 426:45 +CPU= 28.420
  21:35:21 STEP 1708 H= 427:00 +CPU= 29.534
  21:35:50 STEP 1709 H= 427:15 +CPU= 28.494
  21:36:19 STEP 1710 H= 427:30 +CPU= 28.929
  21:36:47 STEP 1711 H= 427:45 +CPU= 28.666
  21:37:17 STEP 1712 H= 428:00 +CPU= 29.274
  21:37:46 STEP 1713 H= 428:15 +CPU= 28.764
  21:38:15 STEP 1714 H= 428:30 +CPU= 28.924
  21:38:44 STEP 1715 H= 428:45 +CPU= 28.785
  21:39:20 STEP 1716 H= 429:00 +CPU= 35.397
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001716
  21:39:48 STEP 1717 H= 429:15 +CPU= 28.421
  21:40:18 STEP 1718 H= 429:30 +CPU= 29.279
  21:40:46 STEP 1719 H= 429:45 +CPU= 28.388
  21:41:16 STEP 1720 H= 430:00 +CPU= 29.758
  21:41:45 STEP 1721 H= 430:15 +CPU= 29.012
  21:42:15 STEP 1722 H= 430:30 +CPU= 29.105
  21:42:44 STEP 1723 H= 430:45 +CPU= 28.665
  21:43:15 STEP 1724 H= 431:00 +CPU= 29.793
  21:43:43 STEP 1725 H= 431:15 +CPU= 28.716
  21:44:12 STEP 1726 H= 431:30 +CPU= 28.718
  21:44:42 STEP 1727 H= 431:45 +CPU= 28.454
  21:45:23 STEP 1728 H= 432:00 +CPU= 39.340
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0_r492272722_17.zip
Uploading the intermediate file: upload_file_17.zip
  21:45:51 STEP 1729 H= 432:15 +CPU= 28.160
  21:46:22 STEP 1730 H= 432:30 +CPU= 27.842
  21:46:51 STEP 1731 H= 432:45 +CPU= 28.503
  21:47:20 STEP 1732 H= 433:00 +CPU= 28.433
  21:47:50 STEP 1733 H= 433:15 +CPU= 28.400
  21:48:19 STEP 1734 H= 433:30 +CPU= 27.983
  21:48:49 STEP 1735 H= 433:45 +CPU= 28.764
  21:49:18 STEP 1736 H= 434:00 +CPU= 29.915
  21:49:48 STEP 1737 H= 434:15 +CPU= 29.111
  21:50:17 STEP 1738 H= 434:30 +CPU= 28.821
  21:50:46 STEP 1739 H= 434:45 +CPU= 29.043
  21:51:22 STEP 1740 H= 435:00 +CPU= 35.111
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001740
  21:51:50 STEP 1741 H= 435:15 +CPU= 28.558
  21:52:20 STEP 1742 H= 435:30 +CPU= 28.749
  21:52:49 STEP 1743 H= 435:45 +CPU= 28.541
  21:53:18 STEP 1744 H= 436:00 +CPU= 29.132
  21:53:47 STEP 1745 H= 436:15 +CPU= 29.166
  21:54:17 STEP 1746 H= 436:30 +CPU= 28.798
  21:54:46 STEP 1747 H= 436:45 +CPU= 28.670
  21:55:17 STEP 1748 H= 437:00 +CPU= 29.678
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3cz
(argv4) batchid: 1018
(argv5) wuid: 12292682
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_49332e72dc041e76616d02f6c56d2cbe to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682.zip
ic_ancil_file: ic_ancil_12292682
ifsdata_file: ifsdata_12292682
climate_data_file: clim_data_12292682
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12292682.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12292682.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4e0347ca0839f52fd2762faaf976f7a7 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12292682.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12292682.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12292682.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12292682.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12292682.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12292682.xml
-- Model is restarting --
Adjusting last_iter, 1748, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12292682.xml
last_cpu_time: 54907
upload_file_number: 18
last_iter: 1729
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0_r492272722
The child process has been launched with process id: 866698
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:866698:866698] [20240614:215705:1718402225:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:57:36 STEP 1728 H= 432:00 +CPU= 27.410
  21:58:04 STEP 1729 H= 432:15 +CPU= 27.498
  21:58:32 STEP 1730 H= 432:30 +CPU= 27.426
  21:59:01 STEP 1731 H= 432:45 +CPU= 28.426
  21:59:30 STEP 1732 H= 433:00 +CPU= 29.140
  21:59:59 STEP 1733 H= 433:15 +CPU= 28.636
  22:00:29 STEP 1734 H= 433:30 +CPU= 28.645
  22:00:58 STEP 1735 H= 433:45 +CPU= 28.811
  22:01:27 STEP 1736 H= 434:00 +CPU= 29.330
  22:01:57 STEP 1737 H= 434:15 +CPU= 29.041
  22:02:26 STEP 1738 H= 434:30 +CPU= 28.450
  22:02:55 STEP 1739 H= 434:45 +CPU= 28.749
  22:03:31 STEP 1740 H= 435:00 +CPU= 35.569
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001740
  22:03:59 STEP 1741 H= 435:15 +CPU= 28.440
  22:04:27 STEP 1742 H= 435:30 +CPU= 27.720
  22:04:56 STEP 1743 H= 435:45 +CPU= 28.410
  22:05:26 STEP 1744 H= 436:00 +CPU= 30.156
  22:05:56 STEP 1745 H= 436:15 +CPU= 28.809
  22:06:23 STEP 1746 H= 436:30 +CPU= 26.235
  22:06:46 STEP 1747 H= 436:45 +CPU= 22.887
  22:07:08 STEP 1748 H= 437:00 +CPU= 21.559
  22:07:29 STEP 1749 H= 437:15 +CPU= 21.646
  22:07:54 STEP 1750 H= 437:30 +CPU= 24.697
  22:08:18 STEP 1751 H= 437:45 +CPU= 24.285
  22:08:50 STEP 1752 H= 438:00 +CPU= 31.655
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001752
  22:09:15 STEP 1753 H= 438:15 +CPU= 24.710
  22:09:43 STEP 1754 H= 438:30 +CPU= 27.734
  22:10:12 STEP 1755 H= 438:45 +CPU= 28.816
  22:10:43 STEP 1756 H= 439:00 +CPU= 29.451
  22:11:12 STEP 1757 H= 439:15 +CPU= 29.130
  22:11:42 STEP 1758 H= 439:30 +CPU= 28.972
  22:12:06 STEP 1759 H= 439:45 +CPU= 23.168
  22:12:26 STEP 1760 H= 440:00 +CPU= 20.478
  22:12:47 STEP 1761 H= 440:15 +CPU= 21.180
  22:13:13 STEP 1762 H= 440:30 +CPU= 25.651
  22:13:35 STEP 1763 H= 440:45 +CPU= 21.862
  22:14:01 STEP 1764 H= 441:00 +CPU= 25.845
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001764
  22:14:26 STEP 1765 H= 441:15 +CPU= 25.313
  22:14:53 STEP 1766 H= 441:30 +CPU= 26.874
  22:15:22 STEP 1767 H= 441:45 +CPU= 28.156
  22:15:50 STEP 1768 H= 442:00 +CPU= 28.560
  22:16:19 STEP 1769 H= 442:15 +CPU= 28.764
  22:16:49 STEP 1770 H= 442:30 +CPU= 28.783
  22:17:18 STEP 1771 H= 442:45 +CPU= 28.876
  22:17:48 STEP 1772 H= 443:00 +CPU= 29.129
  22:18:16 STEP 1773 H= 443:15 +CPU= 26.930
  22:18:37 STEP 1774 H= 443:30 +CPU= 21.518
  22:19:01 STEP 1775 H= 443:45 +CPU= 23.814
  22:19:37 STEP 1776 H= 444:00 +CPU= 35.834
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001776
  22:20:02 STEP 1777 H= 444:15 +CPU= 24.394
  22:20:27 STEP 1778 H= 444:30 +CPU= 24.972
  22:20:51 STEP 1779 H= 444:45 +CPU= 24.154
  22:21:19 STEP 1780 H= 445:00 +CPU= 27.566
  22:21:49 STEP 1781 H= 445:15 +CPU= 28.867
  22:22:18 STEP 1782 H= 445:30 +CPU= 28.764
  22:22:48 STEP 1783 H= 445:45 +CPU= 29.259
  22:23:17 STEP 1784 H= 446:00 +CPU= 29.433
  22:23:47 STEP 1785 H= 446:15 +CPU= 29.200
  22:24:12 STEP 1786 H= 446:30 +CPU= 25.119
  22:24:30 STEP 1787 H= 446:45 +CPU= 17.570
  22:24:53 STEP 1788 H= 447:00 +CPU= 23.148
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001788
  22:25:13 STEP 1789 H= 447:15 +CPU= 20.055
  22:25:38 STEP 1790 H= 447:30 +CPU= 25.060
  22:26:04 STEP 1791 H= 447:45 +CPU= 25.284
  22:26:30 STEP 1792 H= 448:00 +CPU= 26.224
  22:26:55 STEP 1793 H= 448:15 +CPU= 24.134
  22:27:24 STEP 1794 H= 448:30 +CPU= 27.498
  22:27:53 STEP 1795 H= 448:45 +CPU= 29.189
  22:28:24 STEP 1796 H= 449:00 +CPU= 30.092
  22:28:53 STEP 1797 H= 449:15 +CPU= 28.901
  22:29:23 STEP 1798 H= 449:30 +CPU= 28.902
  22:29:52 STEP 1799 H= 449:45 +CPU= 28.636
  22:30:21 STEP 1800 H= 450:00 +CPU= 27.978
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001800
  22:30:43 STEP 1801 H= 450:15 +CPU= 22.345
  22:31:04 STEP 1802 H= 450:30 +CPU= 21.050
  22:31:30 STEP 1803 H= 450:45 +CPU= 26.241
  22:31:55 STEP 1804 H= 451:00 +CPU= 25.041
  22:32:20 STEP 1805 H= 451:15 +CPU= 24.250
  22:32:46 STEP 1806 H= 451:30 +CPU= 26.384
  22:33:13 STEP 1807 H= 451:45 +CPU= 26.699
  22:33:43 STEP 1808 H= 452:00 +CPU= 29.572
  22:34:12 STEP 1809 H= 452:15 +CPU= 28.922
  22:34:41 STEP 1810 H= 452:30 +CPU= 28.872
  22:35:10 STEP 1811 H= 452:45 +CPU= 28.697
  22:35:46 STEP 1812 H= 453:00 +CPU= 35.882
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001812
  22:36:15 STEP 1813 H= 453:15 +CPU= 28.926
  22:36:43 STEP 1814 H= 453:30 +CPU= 28.367
  22:37:12 STEP 1815 H= 453:45 +CPU= 28.796
  22:37:42 STEP 1816 H= 454:00 +CPU= 29.169
  22:38:11 STEP 1817 H= 454:15 +CPU= 29.188
  22:38:40 STEP 1818 H= 454:30 +CPU= 28.520
  22:39:09 STEP 1819 H= 454:45 +CPU= 28.860
  22:39:39 STEP 1820 H= 455:00 +CPU= 29.147
  22:40:08 STEP 1821 H= 455:15 +CPU= 28.975
  22:40:38 STEP 1822 H= 455:30 +CPU= 29.237
  22:41:08 STEP 1823 H= 455:45 +CPU= 29.317
  22:41:44 STEP 1824 H= 456:00 +CPU= 36.593
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0_r492272722_18.zip
Uploading the intermediate file: upload_file_18.zip
  22:42:07 STEP 1825 H= 456:15 +CPU= 22.652
  22:42:28 STEP 1826 H= 456:30 +CPU= 20.962
  22:42:53 STEP 1827 H= 456:45 +CPU= 23.271
  22:43:16 STEP 1828 H= 457:00 +CPU= 22.595
  22:43:41 STEP 1829 H= 457:15 +CPU= 25.617
  22:44:06 STEP 1830 H= 457:30 +CPU= 24.223
  22:44:29 STEP 1831 H= 457:45 +CPU= 23.126
  22:44:58 STEP 1832 H= 458:00 +CPU= 28.532
  22:45:27 STEP 1833 H= 458:15 +CPU= 28.335
  22:45:56 STEP 1834 H= 458:30 +CPU= 28.881
  22:46:25 STEP 1835 H= 458:45 +CPU= 28.667
  22:47:00 STEP 1836 H= 459:00 +CPU= 34.921
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001836
  22:47:30 STEP 1837 H= 459:15 +CPU= 28.804
  22:47:58 STEP 1838 H= 459:30 +CPU= 28.003
  22:48:27 STEP 1839 H= 459:45 +CPU= 28.823
  22:48:57 STEP 1840 H= 460:00 +CPU= 29.001
  22:49:25 STEP 1841 H= 460:15 +CPU= 28.182
  22:49:54 STEP 1842 H= 460:30 +CPU= 28.627
  22:50:23 STEP 1843 H= 460:45 +CPU= 28.705
  22:50:53 STEP 1844 H= 461:00 +CPU= 29.112
  22:51:21 STEP 1845 H= 461:15 +CPU= 28.471
  22:51:50 STEP 1846 H= 461:30 +CPU= 27.862
  22:52:19 STEP 1847 H= 461:45 +CPU= 27.577
  22:52:54 STEP 1848 H= 462:00 +CPU= 34.596
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001848
  22:53:18 STEP 1849 H= 462:15 +CPU= 24.593
  22:53:39 STEP 1850 H= 462:30 +CPU= 20.184
  22:53:59 STEP 1851 H= 462:45 +CPU= 20.017
  22:54:22 STEP 1852 H= 463:00 +CPU= 23.715
  22:54:48 STEP 1853 H= 463:15 +CPU= 25.489
  22:55:08 STEP 1854 H= 463:30 +CPU= 20.447
  22:55:32 STEP 1855 H= 463:45 +CPU= 23.454
  22:55:58 STEP 1856 H= 464:00 +CPU= 25.900
  22:56:26 STEP 1857 H= 464:15 +CPU= 27.965
  22:56:55 STEP 1858 H= 464:30 +CPU= 28.351
  22:57:24 STEP 1859 H= 464:45 +CPU= 28.361
  22:57:59 STEP 1860 H= 465:00 +CPU= 35.168
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001860
  22:58:27 STEP 1861 H= 465:15 +CPU= 27.917
  22:58:56 STEP 1862 H= 465:30 +CPU= 28.307
  22:59:24 STEP 1863 H= 465:45 +CPU= 28.494
  22:59:54 STEP 1864 H= 466:00 +CPU= 29.496
  23:00:23 STEP 1865 H= 466:15 +CPU= 28.372
  23:00:52 STEP 1866 H= 466:30 +CPU= 28.455
  23:01:20 STEP 1867 H= 466:45 +CPU= 28.385
  23:01:49 STEP 1868 H= 467:00 +CPU= 28.597
  23:02:17 STEP 1869 H= 467:15 +CPU= 27.959
  23:02:45 STEP 1870 H= 467:30 +CPU= 27.615
  23:03:13 STEP 1871 H= 467:45 +CPU= 28.258
  23:03:52 STEP 1872 H= 468:00 +CPU= 38.117
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001872
  23:04:20 STEP 1873 H= 468:15 +CPU= 27.748
  23:04:46 STEP 1874 H= 468:30 +CPU= 25.531
  23:05:07 STEP 1875 H= 468:45 +CPU= 20.815
  23:05:30 STEP 1876 H= 469:00 +CPU= 23.418
  23:05:51 STEP 1877 H= 469:15 +CPU= 21.138
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3cz
(argv4) batchid: 1018
(argv5) wuid: 12292682
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/23
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_49332e72dc041e76616d02f6c56d2cbe to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682.zip
ic_ancil_file: ic_ancil_12292682
ifsdata_file: ifsdata_12292682
climate_data_file: clim_data_12292682
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12292682.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12292682.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4e0347ca0839f52fd2762faaf976f7a7 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12292682.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12292682.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12292682.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12292682.zip
Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12292682.xml
Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12292682.xml
-- Model is restarting --
Adjusting last_iter, 1878, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12292682.xml
last_cpu_time: 58995
upload_file_number: 19
last_iter: 1873
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0_r492272722
The child process has been launched with process id: 869419
Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:869419:869419] [20240614:230706:1718406426:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:07:34 STEP 1872 H= 468:00 +CPU= 24.090
  23:08:01 STEP 1873 H= 468:15 +CPU= 26.723
  23:08:30 STEP 1874 H= 468:30 +CPU= 28.431
  23:08:59 STEP 1875 H= 468:45 +CPU= 28.694
  23:09:30 STEP 1876 H= 469:00 +CPU= 29.823
  23:09:59 STEP 1877 H= 469:15 +CPU= 28.624
  23:10:28 STEP 1878 H= 469:30 +CPU= 28.946
  23:10:57 STEP 1879 H= 469:45 +CPU= 29.051
  23:11:24 STEP 1880 H= 470:00 +CPU= 26.970
  23:11:49 STEP 1881 H= 470:15 +CPU= 24.418
  23:12:13 STEP 1882 H= 470:30 +CPU= 24.464
  23:12:40 STEP 1883 H= 470:45 +CPU= 25.995
  23:13:13 STEP 1884 H= 471:00 +CPU= 32.986
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001884
  23:13:39 STEP 1885 H= 471:15 +CPU= 26.302
  23:14:05 STEP 1886 H= 471:30 +CPU= 25.274
  23:14:34 STEP 1887 H= 471:45 +CPU= 28.076
  23:15:04 STEP 1888 H= 472:00 +CPU= 29.632
  23:15:33 STEP 1889 H= 472:15 +CPU= 28.699
  23:16:02 STEP 1890 H= 472:30 +CPU= 28.419
  23:16:31 STEP 1891 H= 472:45 +CPU= 28.491
  23:17:02 STEP 1892 H= 473:00 +CPU= 29.496
  23:17:31 STEP 1893 H= 473:15 +CPU= 28.306
  23:17:54 STEP 1894 H= 473:30 +CPU= 23.024
  23:18:14 STEP 1895 H= 473:45 +CPU= 20.363
  23:18:45 STEP 1896 H= 474:00 +CPU= 31.423
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001896
  23:19:09 STEP 1897 H= 474:15 +CPU= 23.819
  23:19:35 STEP 1898 H= 474:30 +CPU= 26.036
  23:20:01 STEP 1899 H= 474:45 +CPU= 25.480
  23:20:26 STEP 1900 H= 475:00 +CPU= 25.427
  23:20:53 STEP 1901 H= 475:15 +CPU= 26.886
  23:21:23 STEP 1902 H= 475:30 +CPU= 28.839
  23:21:51 STEP 1903 H= 475:45 +CPU= 28.538
  23:22:21 STEP 1904 H= 476:00 +CPU= 29.622
  23:22:50 STEP 1905 H= 476:15 +CPU= 28.775
  23:23:18 STEP 1906 H= 476:30 +CPU= 28.369
  23:23:39 STEP 1907 H= 476:45 +CPU= 20.360
  23:24:03 STEP 1908 H= 477:00 +CPU= 24.580
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001908
  23:24:27 STEP 1909 H= 477:15 +CPU= 23.232
  23:24:52 STEP 1910 H= 477:30 +CPU= 25.678
  23:25:16 STEP 1911 H= 477:45 +CPU= 23.804
  23:25:41 STEP 1912 H= 478:00 +CPU= 24.932
  23:26:09 STEP 1913 H= 478:15 +CPU= 27.134
  23:26:37 STEP 1914 H= 478:30 +CPU= 27.620
  23:27:06 STEP 1915 H= 478:45 +CPU= 28.632
  23:27:35 STEP 1916 H= 479:00 +CPU= 28.907
  23:28:04 STEP 1917 H= 479:15 +CPU= 28.891
  23:28:32 STEP 1918 H= 479:30 +CPU= 28.240
  23:28:58 STEP 1919 H= 479:45 +CPU= 25.669
  23:29:27 STEP 1920 H= 480:00 +CPU= 28.483
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1468238306E-20      I       0.4162640436E-07      I
ICloud fractionI      0.0000000000E+00      I       0.9615649270E-03      I
I  Relat. Hum. I      0.1182807923E+00      I       0.1414595105E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2346337904E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    315923508 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS631847.0 Mbytes/s, TOTAL TIME=    0.0105( 4.8%)
 23:29:27 STEP 1920 H= 480:00 +CPU= 28.483
 END CNT3
 NSTEP =  1920 CNT0   000000000  21.481  21.481  21.528   0.998               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     3.2     1.4     1.8     2.4     3.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1341038.2 1341038.2    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1336882.0 1336882.0     99.69      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    5418.2    5418.2     21.41      0.00
   9 SPCM     - SPECTRAL COMP.                      48     935.5     935.5      3.35      0.00
  10 SCAN2M - PHYSICS                               49   11408.7   11408.7     41.69      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     606.6     606.6      0.18      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      64.9      64.9      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      26.1      26.1      0.01      0.00
  27 RESTART FILES - WRITING                         1    2425.6    2425.6      0.18      0.00
  28 RESTART FILES - READING                         1    3544.9    3544.9      0.26      0.00
  29 SU4FPOS IN CNT4                                 4       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    4740.3    4740.3      1.41      0.00
  31 POSDDH IN STEPO                                 1       9.6       9.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    5677.3    5677.3     20.74      0.00
  39 SU0YOMB                                         1     338.6     338.6      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      28.1      28.1      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     559.4     559.4      2.00      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      69.2      69.2      0.25      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.7       0.7      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     381.9     381.9      3.08      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     234.8     234.8      1.75      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      72.9      72.9      0.54      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     106.4     106.4      0.86      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      58.3      58.3      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      65.3      65.3      0.53      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      36.8      36.8      0.27      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108     130.9     130.9      1.05      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      59.8      59.8      0.45      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1341.0 CPU TIME    1328.8 VECTOR TIME     1328.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1292.6
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/23/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/23/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292682/ICMUAh7zg+001848
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0_r492272722_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
23:32:56 (869416): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Jun 2024 00:07:45 1537687 22446291 oifs_43r3_bl_a3cz_2016092300_20_1018_12292682_0 1,727,100 60,324 0.0349


©2024 cpdn.org