climateprediction.net home page
Task 22276181

Task 22276181

Name oifs_43r3_ps_0900_2004050100_123_973_12190544_0
Workunit 12190544
Created 23 Dec 2022, 9:09:28 UTC
Sent 29 Dec 2022, 16:54:48 UTC
Report deadline 28 Jan 2023, 16:54:48 UTC
Received 2 Jan 2023, 17:03:18 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537562
Run time 11 hours 17 min 10 sec
CPU time 10 hours 34 min 25 sec
Validate state Invalid
Credit 6,773.76
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,788.26 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,962.23 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:8784:8784] [20230101:170830:1672592910:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:08:58 STEP 1560 H=1560:00 +CPU= 25.816
  17:09:13 STEP 1561 H=1561:00 +CPU= 14.783
  17:09:29 STEP 1562 H=1562:00 +CPU= 15.483
  17:09:52 STEP 1563 H=1563:00 +CPU= 22.960
  17:10:09 STEP 1564 H=1564:00 +CPU= 15.405
  17:10:24 STEP 1565 H=1565:00 +CPU= 14.610
  17:10:48 STEP 1566 H=1566:00 +CPU= 23.715
  17:11:03 STEP 1567 H=1567:00 +CPU= 14.857
  17:11:19 STEP 1568 H=1568:00 +CPU= 15.351
  17:11:43 STEP 1569 H=1569:00 +CPU= 23.047
  17:11:58 STEP 1570 H=1570:00 +CPU= 14.973
  17:12:13 STEP 1571 H=1571:00 +CPU= 14.565
  17:12:39 STEP 1572 H=1572:00 +CPU= 25.457
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001572
  17:12:54 STEP 1573 H=1573:00 +CPU= 14.779
  17:13:10 STEP 1574 H=1574:00 +CPU= 15.375
  17:13:33 STEP 1575 H=1575:00 +CPU= 22.910
  17:13:49 STEP 1576 H=1576:00 +CPU= 15.143
  17:14:04 STEP 1577 H=1577:00 +CPU= 14.796
  17:14:28 STEP 1578 H=1578:00 +CPU= 23.381
  17:14:43 STEP 1579 H=1579:00 +CPU= 14.456
  17:14:58 STEP 1580 H=1580:00 +CPU= 15.093
  17:15:22 STEP 1581 H=1581:00 +CPU= 22.797
  17:15:37 STEP 1582 H=1582:00 +CPU= 15.167
  17:15:55 STEP 1583 H=1583:00 +CPU= 15.093
  17:16:24 STEP 1584 H=1584:00 +CPU= 27.170
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_65.zip
Uploading the intermediate file: upload_file_65.zip
  17:16:40 STEP 1585 H=1585:00 +CPU= 14.637
  17:16:55 STEP 1586 H=1586:00 +CPU= 15.115
  17:17:18 STEP 1587 H=1587:00 +CPU= 22.753
  17:17:34 STEP 1588 H=1588:00 +CPU= 15.282
  17:17:49 STEP 1589 H=1589:00 +CPU= 14.655
  17:18:13 STEP 1590 H=1590:00 +CPU= 23.357
  17:18:28 STEP 1591 H=1591:00 +CPU= 14.672
  17:18:44 STEP 1592 H=1592:00 +CPU= 15.029
  17:19:07 STEP 1593 H=1593:00 +CPU= 22.749
  17:19:23 STEP 1594 H=1594:00 +CPU= 15.313
  17:19:38 STEP 1595 H=1595:00 +CPU= 14.685
  17:20:04 STEP 1596 H=1596:00 +CPU= 25.644
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001596
  17:20:19 STEP 1597 H=1597:00 +CPU= 14.315
  17:20:34 STEP 1598 H=1598:00 +CPU= 14.325
  17:20:55 STEP 1599 H=1599:00 +CPU= 21.537
  17:21:11 STEP 1600 H=1600:00 +CPU= 14.720
  17:21:25 STEP 1601 H=1601:00 +CPU= 14.289
  17:21:49 STEP 1602 H=1602:00 +CPU= 22.974
  17:22:04 STEP 1603 H=1603:00 +CPU= 14.253
  17:22:19 STEP 1604 H=1604:00 +CPU= 14.849
  17:22:42 STEP 1605 H=1605:00 +CPU= 22.413
  17:22:57 STEP 1606 H=1606:00 +CPU= 14.727
  17:23:12 STEP 1607 H=1607:00 +CPU= 14.731
  17:23:40 STEP 1608 H=1608:00 +CPU= 26.459
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_66.zip
Uploading the intermediate file: upload_file_66.zip
  17:23:55 STEP 1609 H=1609:00 +CPU= 14.359
  17:24:10 STEP 1610 H=1610:00 +CPU= 14.887
  17:24:33 STEP 1611 H=1611:00 +CPU= 22.482
  17:24:48 STEP 1612 H=1612:00 +CPU= 14.871
  17:25:03 STEP 1613 H=1613:00 +CPU= 14.324
  17:25:27 STEP 1614 H=1614:00 +CPU= 23.265
  17:25:42 STEP 1615 H=1615:00 +CPU= 14.413
  17:25:57 STEP 1616 H=1616:00 +CPU= 14.867
  17:26:20 STEP 1617 H=1617:00 +CPU= 22.702
  17:26:35 STEP 1618 H=1618:00 +CPU= 14.883
  17:26:50 STEP 1619 H=1619:00 +CPU= 14.174
  17:27:16 STEP 1620 H=1620:00 +CPU= 25.461
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001620
  17:27:31 STEP 1621 H=1621:00 +CPU= 14.423
  17:27:46 STEP 1622 H=1622:00 +CPU= 14.912
  17:28:09 STEP 1623 H=1623:00 +CPU= 22.549
  17:28:24 STEP 1624 H=1624:00 +CPU= 14.811
  17:28:39 STEP 1625 H=1625:00 +CPU= 14.391
  17:29:02 STEP 1626 H=1626:00 +CPU= 23.230
  17:29:16 STEP 1627 H=1627:00 +CPU= 14.224
  17:29:31 STEP 1628 H=1628:00 +CPU= 14.701
  17:29:54 STEP 1629 H=1629:00 +CPU= 22.530
  17:30:09 STEP 1630 H=1630:00 +CPU= 14.642
  17:30:24 STEP 1631 H=1631:00 +CPU= 14.607
  17:30:50 STEP 1632 H=1632:00 +CPU= 26.150
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_67.zip
Uploading the intermediate file: upload_file_67.zip
  17:31:05 STEP 1633 H=1633:00 +CPU= 14.244
  17:31:20 STEP 1634 H=1634:00 +CPU= 14.732
  17:31:42 STEP 1635 H=1635:00 +CPU= 22.286
  17:31:57 STEP 1636 H=1636:00 +CPU= 14.765
  17:32:11 STEP 1637 H=1637:00 +CPU= 14.257
  17:32:35 STEP 1638 H=1638:00 +CPU= 23.108
  17:32:49 STEP 1639 H=1639:00 +CPU= 14.373
  17:33:04 STEP 1640 H=1640:00 +CPU= 14.615
  17:33:27 STEP 1641 H=1641:00 +CPU= 22.443
  17:33:42 STEP 1642 H=1642:00 +CPU= 14.747
  17:33:57 STEP 1643 H=1643:00 +CPU= 14.304
  17:34:23 STEP 1644 H=1644:00 +CPU= 25.682
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001644
  17:34:38 STEP 1645 H=1645:00 +CPU= 14.160
  17:34:53 STEP 1646 H=1646:00 +CPU= 14.884
  17:35:15 STEP 1647 H=1647:00 +CPU= 22.204
  17:35:30 STEP 1648 H=1648:00 +CPU= 14.766
  17:35:45 STEP 1649 H=1649:00 +CPU= 14.320
  17:36:08 STEP 1650 H=1650:00 +CPU= 23.151
  17:36:22 STEP 1651 H=1651:00 +CPU= 14.198
  17:36:37 STEP 1652 H=1652:00 +CPU= 14.795
  17:37:00 STEP 1653 H=1653:00 +CPU= 22.574
  17:37:15 STEP 1654 H=1654:00 +CPU= 14.815
  17:37:31 STEP 1655 H=1655:00 +CPU= 14.542
  17:38:00 STEP 1656 H=1656:00 +CPU= 26.444
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_68.zip
Uploading the intermediate file: upload_file_68.zip
  17:38:14 STEP 1657 H=1657:00 +CPU= 14.270
  17:38:29 STEP 1658 H=1658:00 +CPU= 14.920
  17:38:52 STEP 1659 H=1659:00 +CPU= 22.711
  17:39:07 STEP 1660 H=1660:00 +CPU= 14.913
  17:39:22 STEP 1661 H=1661:00 +CPU= 14.155
  17:39:45 STEP 1662 H=1662:00 +CPU= 23.100
  17:39:59 STEP 1663 H=1663:00 +CPU= 14.349
  17:40:14 STEP 1664 H=1664:00 +CPU= 14.851
  17:40:37 STEP 1665 H=1665:00 +CPU= 22.676
  17:40:52 STEP 1666 H=1666:00 +CPU= 14.956
  17:41:06 STEP 1667 H=1667:00 +CPU= 14.223
  17:41:32 STEP 1668 H=1668:00 +CPU= 25.564
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001668
  17:41:47 STEP 1669 H=1669:00 +CPU= 14.375
  17:42:02 STEP 1670 H=1670:00 +CPU= 14.895
  17:42:24 STEP 1671 H=1671:00 +CPU= 22.595
  17:42:39 STEP 1672 H=1672:00 +CPU= 14.863
  17:42:54 STEP 1673 H=1673:00 +CPU= 14.174
  17:43:17 STEP 1674 H=1674:00 +CPU= 23.060
  17:43:31 STEP 1675 H=1675:00 +CPU= 14.235
  17:43:46 STEP 1676 H=1676:00 +CPU= 14.666
  17:44:08 STEP 1677 H=1677:00 +CPU= 22.230
  17:44:23 STEP 1678 H=1678:00 +CPU= 14.713
  17:44:38 STEP 1679 H=1679:00 +CPU= 14.368
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0900
(argv4) batchid: 973
(argv5) wuid: 12190544
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a82d3dc5a58b5afbc9877e03cf8c3d22 to: /var/lib/boinc/slots/16/oifs_43r3_ps_0900_2004050100_123_973_12190544.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0900_2004050100_123_973_12190544.zip
ic_ancil_file: ic_ancil_12190544
ifsdata_file: ifsdata_12190544
climate_data_file: clim_data_12190544
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_27b817844710421480f75c19f86f137f to: /var/lib/boinc/slots/16/ic_ancil_12190544.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12190544.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b11f49fc573c9afb90faa52bb9483ffe to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12190544.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12190544.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12190544.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12190544.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12190544.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12190544.xml
-- Model is restarting --
Adjusting last_iter, 1680, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12190544.xml
last_cpu_time: 34416
upload_file_number: 69
last_iter: 1679
last_upload: 5961600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502
The child process has been launched with process id: 11504
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:11504:11504] [20230101:190452:1672599892:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:05:19 STEP 1680 H=1680:00 +CPU= 25.272
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_69.zip
Uploading the intermediate file: upload_file_69.zip
  19:05:34 STEP 1681 H=1681:00 +CPU= 14.384
  19:05:49 STEP 1682 H=1682:00 +CPU= 14.657
  19:06:12 STEP 1683 H=1683:00 +CPU= 22.649
  19:06:26 STEP 1684 H=1684:00 +CPU= 14.742
  19:06:41 STEP 1685 H=1685:00 +CPU= 14.106
  19:07:04 STEP 1686 H=1686:00 +CPU= 23.111
  19:07:19 STEP 1687 H=1687:00 +CPU= 14.394
  19:07:34 STEP 1688 H=1688:00 +CPU= 14.903
  19:07:57 STEP 1689 H=1689:00 +CPU= 22.812
  19:08:13 STEP 1690 H=1690:00 +CPU= 14.788
  19:08:28 STEP 1691 H=1691:00 +CPU= 14.283
  19:08:55 STEP 1692 H=1692:00 +CPU= 25.512
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001692
  19:09:17 STEP 1693 H=1693:00 +CPU= 14.511
  19:09:33 STEP 1694 H=1694:00 +CPU= 15.055
  19:09:58 STEP 1695 H=1695:00 +CPU= 22.687
  19:10:14 STEP 1696 H=1696:00 +CPU= 14.935
  19:10:29 STEP 1697 H=1697:00 +CPU= 14.284
  19:10:54 STEP 1698 H=1698:00 +CPU= 23.378
  19:11:08 STEP 1699 H=1699:00 +CPU= 14.214
  19:11:23 STEP 1700 H=1700:00 +CPU= 14.886
  19:11:46 STEP 1701 H=1701:00 +CPU= 22.465
  19:12:01 STEP 1702 H=1702:00 +CPU= 14.857
  19:12:16 STEP 1703 H=1703:00 +CPU= 14.620
  19:12:45 STEP 1704 H=1704:00 +CPU= 26.233
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_70.zip
Uploading the intermediate file: upload_file_70.zip
  19:13:00 STEP 1705 H=1705:00 +CPU= 14.366
  19:13:15 STEP 1706 H=1706:00 +CPU= 14.945
  19:13:38 STEP 1707 H=1707:00 +CPU= 22.681
  19:13:54 STEP 1708 H=1708:00 +CPU= 14.809
  19:14:08 STEP 1709 H=1709:00 +CPU= 14.413
  19:14:33 STEP 1710 H=1710:00 +CPU= 23.257
  19:14:48 STEP 1711 H=1711:00 +CPU= 14.388
  19:15:05 STEP 1712 H=1712:00 +CPU= 14.800
  19:15:27 STEP 1713 H=1713:00 +CPU= 22.465
  19:15:43 STEP 1714 H=1714:00 +CPU= 14.894
  19:15:58 STEP 1715 H=1715:00 +CPU= 14.436
  19:16:23 STEP 1716 H=1716:00 +CPU= 25.355
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001716
  19:16:38 STEP 1717 H=1717:00 +CPU= 14.458
  19:16:53 STEP 1718 H=1718:00 +CPU= 14.867
  19:17:16 STEP 1719 H=1719:00 +CPU= 22.697
  19:17:32 STEP 1720 H=1720:00 +CPU= 14.925
  19:17:46 STEP 1721 H=1721:00 +CPU= 14.372
  19:18:10 STEP 1722 H=1722:00 +CPU= 23.341
  19:18:26 STEP 1723 H=1723:00 +CPU= 14.026
  19:18:46 STEP 1724 H=1724:00 +CPU= 14.441
  19:19:26 STEP 1725 H=1725:00 +CPU= 21.951
  19:19:41 STEP 1726 H=1726:00 +CPU= 14.856
  19:19:56 STEP 1727 H=1727:00 +CPU= 14.732
  19:20:23 STEP 1728 H=1728:00 +CPU= 26.011
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_71.zip
Uploading the intermediate file: upload_file_71.zip
  19:20:37 STEP 1729 H=1729:00 +CPU= 14.242
  19:21:09 STEP 1730 H=1730:00 +CPU= 14.679
  19:21:33 STEP 1731 H=1731:00 +CPU= 22.472
  19:21:47 STEP 1732 H=1732:00 +CPU= 14.521
  19:22:03 STEP 1733 H=1733:00 +CPU= 14.195
  19:22:28 STEP 1734 H=1734:00 +CPU= 23.299
  19:22:43 STEP 1735 H=1735:00 +CPU= 13.519
  19:22:59 STEP 1736 H=1736:00 +CPU= 14.075
  19:23:24 STEP 1737 H=1737:00 +CPU= 21.651
  19:23:40 STEP 1738 H=1738:00 +CPU= 13.725
  19:23:56 STEP 1739 H=1739:00 +CPU= 13.808
  19:24:23 STEP 1740 H=1740:00 +CPU= 24.924
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001740
  19:24:40 STEP 1741 H=1741:00 +CPU= 13.863
  19:24:57 STEP 1742 H=1742:00 +CPU= 14.123
  19:25:23 STEP 1743 H=1743:00 +CPU= 21.390
  19:25:40 STEP 1744 H=1744:00 +CPU= 14.314
  19:25:56 STEP 1745 H=1745:00 +CPU= 14.002
  19:26:22 STEP 1746 H=1746:00 +CPU= 22.385
  19:26:38 STEP 1747 H=1747:00 +CPU= 14.204
  19:26:53 STEP 1748 H=1748:00 +CPU= 14.523
  19:27:18 STEP 1749 H=1749:00 +CPU= 21.634
  19:27:34 STEP 1750 H=1750:00 +CPU= 14.260
  19:27:50 STEP 1751 H=1751:00 +CPU= 14.237
  19:28:17 STEP 1752 H=1752:00 +CPU= 26.172
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_72.zip
Uploading the intermediate file: upload_file_72.zip
  19:28:35 STEP 1753 H=1753:00 +CPU= 14.141
  19:28:52 STEP 1754 H=1754:00 +CPU= 14.865
  19:29:15 STEP 1755 H=1755:00 +CPU= 22.129
  19:29:30 STEP 1756 H=1756:00 +CPU= 14.914
  19:29:47 STEP 1757 H=1757:00 +CPU= 14.229
  19:30:11 STEP 1758 H=1758:00 +CPU= 23.105
  19:30:26 STEP 1759 H=1759:00 +CPU= 14.028
  19:30:43 STEP 1760 H=1760:00 +CPU= 14.644
  19:31:06 STEP 1761 H=1761:00 +CPU= 22.496
  19:31:28 STEP 1762 H=1762:00 +CPU= 14.487
  19:31:45 STEP 1763 H=1763:00 +CPU= 14.572
  19:32:11 STEP 1764 H=1764:00 +CPU= 25.437
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001764
  19:32:27 STEP 1765 H=1765:00 +CPU= 14.262
  19:32:57 STEP 1766 H=1766:00 +CPU= 14.997
  19:33:20 STEP 1767 H=1767:00 +CPU= 22.276
  19:33:37 STEP 1768 H=1768:00 +CPU= 14.866
  19:33:52 STEP 1769 H=1769:00 +CPU= 14.358
  19:34:22 STEP 1770 H=1770:00 +CPU= 23.220
  19:34:38 STEP 1771 H=1771:00 +CPU= 14.347
  19:34:53 STEP 1772 H=1772:00 +CPU= 14.350
  19:35:28 STEP 1773 H=1773:00 +CPU= 22.357
  19:35:45 STEP 1774 H=1774:00 +CPU= 14.487
  19:36:01 STEP 1775 H=1775:00 +CPU= 14.843
  19:36:38 STEP 1776 H=1776:00 +CPU= 26.378
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_73.zip
Uploading the intermediate file: upload_file_73.zip
  19:37:15 STEP 1777 H=1777:00 +CPU= 13.905
  19:37:31 STEP 1778 H=1778:00 +CPU= 14.440
  19:37:55 STEP 1779 H=1779:00 +CPU= 22.699
  19:38:21 STEP 1780 H=1780:00 +CPU= 14.534
  19:38:37 STEP 1781 H=1781:00 +CPU= 14.092
  19:39:02 STEP 1782 H=1782:00 +CPU= 22.850
  19:39:18 STEP 1783 H=1783:00 +CPU= 14.494
  19:40:09 STEP 1784 H=1784:00 +CPU= 14.332
  19:40:31 STEP 1785 H=1785:00 +CPU= 21.327
  19:40:48 STEP 1786 H=1786:00 +CPU= 14.704
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0900
(argv4) batchid: 973
(argv5) wuid: 12190544
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a82d3dc5a58b5afbc9877e03cf8c3d22 to: /var/lib/boinc/slots/16/oifs_43r3_ps_0900_2004050100_123_973_12190544.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0900_2004050100_123_973_12190544.zip
ic_ancil_file: ic_ancil_12190544
ifsdata_file: ifsdata_12190544
climate_data_file: clim_data_12190544
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_27b817844710421480f75c19f86f137f to: /var/lib/boinc/slots/16/ic_ancil_12190544.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12190544.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b11f49fc573c9afb90faa52bb9483ffe to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12190544.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12190544.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12190544.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12190544.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12190544.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12190544.xml
-- Model is restarting --
Adjusting last_iter, 1787, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12190544.xml
last_cpu_time: 36289
upload_file_number: 74
last_iter: 1775
last_upload: 6393600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502
The child process has been launched with process id: 15637
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:15637:15637] [20230101:213626:1672608986:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:37:19 STEP 1776 H=1776:00 +CPU= 25.462
  21:37:34 STEP 1777 H=1777:00 +CPU= 14.752
  21:37:50 STEP 1778 H=1778:00 +CPU= 15.258
  21:38:13 STEP 1779 H=1779:00 +CPU= 23.369
  21:38:29 STEP 1780 H=1780:00 +CPU= 15.324
  21:38:43 STEP 1781 H=1781:00 +CPU= 14.578
  21:39:07 STEP 1782 H=1782:00 +CPU= 23.873
  21:39:22 STEP 1783 H=1783:00 +CPU= 14.645
  21:39:38 STEP 1784 H=1784:00 +CPU= 15.312
  21:40:01 STEP 1785 H=1785:00 +CPU= 23.039
  21:40:16 STEP 1786 H=1786:00 +CPU= 15.337
  21:40:31 STEP 1787 H=1787:00 +CPU= 14.696
  21:40:57 STEP 1788 H=1788:00 +CPU= 25.837
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001788
  21:41:12 STEP 1789 H=1789:00 +CPU= 14.746
  21:41:27 STEP 1790 H=1790:00 +CPU= 15.144
  21:41:50 STEP 1791 H=1791:00 +CPU= 22.744
  21:42:05 STEP 1792 H=1792:00 +CPU= 15.039
  21:42:20 STEP 1793 H=1793:00 +CPU= 14.414
  21:42:43 STEP 1794 H=1794:00 +CPU= 23.386
  21:42:58 STEP 1795 H=1795:00 +CPU= 14.514
  21:43:13 STEP 1796 H=1796:00 +CPU= 14.971
  21:43:36 STEP 1797 H=1797:00 +CPU= 22.534
  21:43:51 STEP 1798 H=1798:00 +CPU= 14.912
  21:44:05 STEP 1799 H=1799:00 +CPU= 14.857
  21:44:33 STEP 1800 H=1800:00 +CPU= 26.588
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_74.zip
Uploading the intermediate file: upload_file_74.zip
  21:44:47 STEP 1801 H=1801:00 +CPU= 14.417
  21:45:02 STEP 1802 H=1802:00 +CPU= 14.771
  21:45:25 STEP 1803 H=1803:00 +CPU= 22.786
  21:45:40 STEP 1804 H=1804:00 +CPU= 14.811
  21:45:52 STEP 1805 H=1805:00 +CPU= 11.792
  21:46:11 STEP 1806 H=1806:00 +CPU= 19.242
  21:46:23 STEP 1807 H=1807:00 +CPU= 12.001
  21:46:35 STEP 1808 H=1808:00 +CPU= 11.963
  21:46:57 STEP 1809 H=1809:00 +CPU= 21.923
  21:47:12 STEP 1810 H=1810:00 +CPU= 14.405
  21:47:26 STEP 1811 H=1811:00 +CPU= 13.856
  21:47:51 STEP 1812 H=1812:00 +CPU= 24.777
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001812
  21:48:05 STEP 1813 H=1813:00 +CPU= 13.873
  21:48:19 STEP 1814 H=1814:00 +CPU= 14.398
  21:48:41 STEP 1815 H=1815:00 +CPU= 21.929
  21:48:56 STEP 1816 H=1816:00 +CPU= 14.399
  21:49:10 STEP 1817 H=1817:00 +CPU= 13.846
  21:49:32 STEP 1818 H=1818:00 +CPU= 22.513
  21:49:46 STEP 1819 H=1819:00 +CPU= 13.814
  21:50:00 STEP 1820 H=1820:00 +CPU= 14.368
  21:50:22 STEP 1821 H=1821:00 +CPU= 21.872
  21:50:37 STEP 1822 H=1822:00 +CPU= 14.346
  21:50:51 STEP 1823 H=1823:00 +CPU= 14.264
  21:51:17 STEP 1824 H=1824:00 +CPU= 25.680
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_75.zip
Uploading the intermediate file: upload_file_75.zip
  21:51:31 STEP 1825 H=1825:00 +CPU= 13.842
  21:51:45 STEP 1826 H=1826:00 +CPU= 14.354
  21:52:07 STEP 1827 H=1827:00 +CPU= 21.899
  21:52:22 STEP 1828 H=1828:00 +CPU= 14.378
  21:52:36 STEP 1829 H=1829:00 +CPU= 13.817
  21:52:58 STEP 1830 H=1830:00 +CPU= 22.531
  21:53:12 STEP 1831 H=1831:00 +CPU= 13.831
  21:53:27 STEP 1832 H=1832:00 +CPU= 14.243
  21:53:48 STEP 1833 H=1833:00 +CPU= 21.896
  21:54:03 STEP 1834 H=1834:00 +CPU= 14.393
  21:54:17 STEP 1835 H=1835:00 +CPU= 13.856
  21:54:42 STEP 1836 H=1836:00 +CPU= 24.754
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001836
  21:54:56 STEP 1837 H=1837:00 +CPU= 13.859
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0900
(argv4) batchid: 973
(argv5) wuid: 12190544
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a82d3dc5a58b5afbc9877e03cf8c3d22 to: /var/lib/boinc/slots/16/oifs_43r3_ps_0900_2004050100_123_973_12190544.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0900_2004050100_123_973_12190544.zip
ic_ancil_file: ic_ancil_12190544
ifsdata_file: ifsdata_12190544
climate_data_file: clim_data_12190544
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_27b817844710421480f75c19f86f137f to: /var/lib/boinc/slots/16/ic_ancil_12190544.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12190544.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b11f49fc573c9afb90faa52bb9483ffe to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12190544.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12190544.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12190544.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12190544.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12190544.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12190544.xml
-- Model is restarting --
Adjusting last_iter, 1838, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12190544.xml
last_cpu_time: 37370
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502
The child process has been launched with process id: 14226
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:14226:14226] [20230102:155122:1672674682:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:51:53 STEP 1824 H=1824:00 +CPU= 24.358
  15:52:06 STEP 1825 H=1825:00 +CPU= 13.711
  15:52:21 STEP 1826 H=1826:00 +CPU= 14.264
  15:52:43 STEP 1827 H=1827:00 +CPU= 21.692
  15:52:57 STEP 1828 H=1828:00 +CPU= 14.257
  15:53:11 STEP 1829 H=1829:00 +CPU= 13.705
  15:53:33 STEP 1830 H=1830:00 +CPU= 22.353
  15:53:47 STEP 1831 H=1831:00 +CPU= 13.742
  15:54:01 STEP 1832 H=1832:00 +CPU= 14.294
  15:54:23 STEP 1833 H=1833:00 +CPU= 21.732
  15:54:37 STEP 1834 H=1834:00 +CPU= 14.289
  15:54:51 STEP 1835 H=1835:00 +CPU= 13.729
  15:55:16 STEP 1836 H=1836:00 +CPU= 24.540
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001836
  15:55:30 STEP 1837 H=1837:00 +CPU= 13.769
  15:55:44 STEP 1838 H=1838:00 +CPU= 14.326
  15:56:06 STEP 1839 H=1839:00 +CPU= 21.730
  15:56:20 STEP 1840 H=1840:00 +CPU= 14.298
  15:56:34 STEP 1841 H=1841:00 +CPU= 13.767
  15:56:56 STEP 1842 H=1842:00 +CPU= 22.386
  15:57:10 STEP 1843 H=1843:00 +CPU= 13.752
  15:57:25 STEP 1844 H=1844:00 +CPU= 14.294
  15:57:46 STEP 1845 H=1845:00 +CPU= 21.728
  15:58:01 STEP 1846 H=1846:00 +CPU= 14.287
  15:58:15 STEP 1847 H=1847:00 +CPU= 14.192
  15:58:41 STEP 1848 H=1848:00 +CPU= 25.469
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12190544/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0900_2004050100_123_973_12190544_0_r527224502_76.zip
Uploading the intermediate file: upload_file_76.zip
  15:58:54 STEP 1849 H=1849:00 +CPU= 13.747
  15:59:09 STEP 1850 H=1850:00 +CPU= 14.269
  15:59:31 STEP 1851 H=1851:00 +CPU= 21.727
  15:59:45 STEP 1852 H=1852:00 +CPU= 14.294
  15:59:59 STEP 1853 H=1853:00 +CPU= 13.713
  16:00:21 STEP 1854 H=1854:00 +CPU= 22.367
  16:00:35 STEP 1855 H=1855:00 +CPU= 13.774
  16:00:50 STEP 1856 H=1856:00 +CPU= 14.313
  16:01:12 STEP 1857 H=1857:00 +CPU= 21.772
  16:01:26 STEP 1858 H=1858:00 +CPU= 14.064
  16:01:40 STEP 1859 H=1859:00 +CPU= 13.779
  16:02:04 STEP 1860 H=1860:00 +CPU= 24.562
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001860
  16:02:18 STEP 1861 H=1861:00 +CPU= 13.778
  16:02:32 STEP 1862 H=1862:00 +CPU= 14.124
  16:02:54 STEP 1863 H=1863:00 +CPU= 21.178
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:07 1537562 22276181 oifs_43r3_ps_0900_2004050100_123_973_12190544_0 4,838,400 27,638 0.0057
09 May 2023 16:03:03 1537562 22276181 oifs_43r3_ps_0900_2004050100_123_973_12190544_0 3,974,400 22,963 0.0058
09 May 2023 16:02:51 1537562 22276181 oifs_43r3_ps_0900_2004050100_123_973_12190544_0 3,110,400 17,797 0.0057
09 May 2023 16:02:07 1537562 22276181 oifs_43r3_ps_0900_2004050100_123_973_12190544_0 1,728,000 9,795 0.0057
09 May 2023 16:02:03 1537562 22276181 oifs_43r3_ps_0900_2004050100_123_973_12190544_0 864,000 4,965 0.0057


©2024 climateprediction.net