climateprediction.net home page
Task 22269739

Task 22269739

Name oifs_43r3_ps_0838_1998050100_123_967_12184482_0
Workunit 12184482
Created 22 Dec 2022, 17:14:52 UTC
Sent 15 Jan 2023, 2:24:06 UTC
Report deadline 14 Feb 2023, 2:24:06 UTC
Received 15 Jan 2023, 18:57:59 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1500195
Run time 14 hours 44 min 54 sec
CPU time 14 hours 18 min 35 sec
Validate state Invalid
Credit 4,838.40
Device peak FLOPS 5.54 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.17 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,688.43 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1676990:1676990] [20230115:145941:1673791181:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:00:34 STEP 1536 H=1536:00 +CPU= 45.597
  15:01:01 STEP 1537 H=1537:00 +CPU= 25.305
  15:01:31 STEP 1538 H=1538:00 +CPU= 28.810
  15:02:12 STEP 1539 H=1539:00 +CPU= 39.060
  15:02:40 STEP 1540 H=1540:00 +CPU= 26.784
  15:03:08 STEP 1541 H=1541:00 +CPU= 27.614
  15:03:49 STEP 1542 H=1542:00 +CPU= 39.630
  15:04:17 STEP 1543 H=1543:00 +CPU= 26.004
  15:04:45 STEP 1544 H=1544:00 +CPU= 26.756
  15:05:25 STEP 1545 H=1545:00 +CPU= 39.099
  15:05:55 STEP 1546 H=1546:00 +CPU= 27.404
  15:06:26 STEP 1547 H=1547:00 +CPU= 26.470
  15:07:18 STEP 1548 H=1548:00 +CPU= 45.293
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001548
  15:07:49 STEP 1549 H=1549:00 +CPU= 25.403
  15:08:20 STEP 1550 H=1550:00 +CPU= 28.364
  15:09:03 STEP 1551 H=1551:00 +CPU= 39.635
  15:09:30 STEP 1552 H=1552:00 +CPU= 26.291
  15:10:00 STEP 1553 H=1553:00 +CPU= 26.809
  15:10:43 STEP 1554 H=1554:00 +CPU= 40.098
  15:11:12 STEP 1555 H=1555:00 +CPU= 26.905
  15:11:42 STEP 1556 H=1556:00 +CPU= 27.240
  15:12:23 STEP 1557 H=1557:00 +CPU= 39.203
  15:12:53 STEP 1558 H=1558:00 +CPU= 27.389
  15:13:25 STEP 1559 H=1559:00 +CPU= 27.751
  15:14:30 STEP 1560 H=1560:00 +CPU= 48.616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_64.zip
Uploading the intermediate file: upload_file_64.zip
  15:14:59 STEP 1561 H=1561:00 +CPU= 26.925
  15:15:27 STEP 1562 H=1562:00 +CPU= 26.630
  15:16:09 STEP 1563 H=1563:00 +CPU= 39.633
  15:16:41 STEP 1564 H=1564:00 +CPU= 28.155
  15:17:08 STEP 1565 H=1565:00 +CPU= 25.990
  15:17:52 STEP 1566 H=1566:00 +CPU= 40.709
  15:18:23 STEP 1567 H=1567:00 +CPU= 27.859
  15:18:51 STEP 1568 H=1568:00 +CPU= 26.496
  15:19:34 STEP 1569 H=1569:00 +CPU= 40.035
  15:20:05 STEP 1570 H=1570:00 +CPU= 28.357
  15:20:32 STEP 1571 H=1571:00 +CPU= 26.036
  15:21:21 STEP 1572 H=1572:00 +CPU= 44.670
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001572
  15:21:50 STEP 1573 H=1573:00 +CPU= 26.822
  15:22:18 STEP 1574 H=1574:00 +CPU= 26.083
  15:23:01 STEP 1575 H=1575:00 +CPU= 39.995
  15:23:31 STEP 1576 H=1576:00 +CPU= 27.344
  15:23:58 STEP 1577 H=1577:00 +CPU= 25.853
  15:24:43 STEP 1578 H=1578:00 +CPU= 41.421
  15:25:10 STEP 1579 H=1579:00 +CPU= 25.869
  15:25:38 STEP 1580 H=1580:00 +CPU= 26.474
  15:26:24 STEP 1581 H=1581:00 +CPU= 41.248
  15:26:52 STEP 1582 H=1582:00 +CPU= 26.914
  15:27:21 STEP 1583 H=1583:00 +CPU= 26.818
  15:28:15 STEP 1584 H=1584:00 +CPU= 48.767
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_65.zip
Uploading the intermediate file: upload_file_65.zip
  15:28:43 STEP 1585 H=1585:00 +CPU= 26.389
  15:29:13 STEP 1586 H=1586:00 +CPU= 28.288
  15:29:55 STEP 1587 H=1587:00 +CPU= 39.563
  15:30:24 STEP 1588 H=1588:00 +CPU= 27.300
  15:30:55 STEP 1589 H=1589:00 +CPU= 28.086
  15:31:37 STEP 1590 H=1590:00 +CPU= 40.175
  15:32:04 STEP 1591 H=1591:00 +CPU= 26.083
  15:32:34 STEP 1592 H=1592:00 +CPU= 27.885
  15:33:15 STEP 1593 H=1593:00 +CPU= 38.944
  15:33:45 STEP 1594 H=1594:00 +CPU= 28.472
  15:34:14 STEP 1595 H=1595:00 +CPU= 26.468
  15:34:59 STEP 1596 H=1596:00 +CPU= 43.497
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001596
  15:35:30 STEP 1597 H=1597:00 +CPU= 27.611
  15:35:59 STEP 1598 H=1598:00 +CPU= 26.937
  15:36:40 STEP 1599 H=1599:00 +CPU= 39.056
  15:37:12 STEP 1600 H=1600:00 +CPU= 28.666
  15:37:39 STEP 1601 H=1601:00 +CPU= 26.056
  15:38:22 STEP 1602 H=1602:00 +CPU= 39.862
  15:38:53 STEP 1603 H=1603:00 +CPU= 27.942
  15:39:22 STEP 1604 H=1604:00 +CPU= 26.986
  15:40:05 STEP 1605 H=1605:00 +CPU= 39.939
  15:40:36 STEP 1606 H=1606:00 +CPU= 28.335
  15:41:07 STEP 1607 H=1607:00 +CPU= 28.003
  15:41:55 STEP 1608 H=1608:00 +CPU= 46.139
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_66.zip
Uploading the intermediate file: upload_file_66.zip
  15:42:25 STEP 1609 H=1609:00 +CPU= 27.205
  15:42:53 STEP 1610 H=1610:00 +CPU= 26.409
  15:43:34 STEP 1611 H=1611:00 +CPU= 39.323
  15:44:02 STEP 1612 H=1612:00 +CPU= 26.147
  15:44:28 STEP 1613 H=1613:00 +CPU= 25.331
  15:45:12 STEP 1614 H=1614:00 +CPU= 41.109
  15:45:39 STEP 1615 H=1615:00 +CPU= 25.501
  15:46:08 STEP 1616 H=1616:00 +CPU= 27.330
  15:46:51 STEP 1617 H=1617:00 +CPU= 39.583
  15:47:19 STEP 1618 H=1618:00 +CPU= 26.706
  15:47:46 STEP 1619 H=1619:00 +CPU= 26.088
  15:48:33 STEP 1620 H=1620:00 +CPU= 43.956
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001620
  15:49:00 STEP 1621 H=1621:00 +CPU= 26.210
  15:49:30 STEP 1622 H=1622:00 +CPU= 27.416
  15:50:18 STEP 1623 H=1623:00 +CPU= 41.361
  15:50:51 STEP 1624 H=1624:00 +CPU= 28.332
  15:51:18 STEP 1625 H=1625:00 +CPU= 25.640
  15:52:04 STEP 1626 H=1626:00 +CPU= 41.762
  15:52:31 STEP 1627 H=1627:00 +CPU= 25.765
  15:53:03 STEP 1628 H=1628:00 +CPU= 28.064
  15:53:44 STEP 1629 H=1629:00 +CPU= 38.964
  15:54:13 STEP 1630 H=1630:00 +CPU= 26.906
  15:54:42 STEP 1631 H=1631:00 +CPU= 27.300
  15:55:31 STEP 1632 H=1632:00 +CPU= 46.060
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_67.zip
Uploading the intermediate file: upload_file_67.zip
  15:55:59 STEP 1633 H=1633:00 +CPU= 26.256
  15:56:27 STEP 1634 H=1634:00 +CPU= 26.356
  15:57:08 STEP 1635 H=1635:00 +CPU= 38.911
  15:57:35 STEP 1636 H=1636:00 +CPU= 26.159
  15:58:04 STEP 1637 H=1637:00 +CPU= 26.687
  15:58:47 STEP 1638 H=1638:00 +CPU= 40.245
  15:59:14 STEP 1639 H=1639:00 +CPU= 26.099
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0838
(argv4) batchid: 967
(argv5) wuid: 12184482
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a80facd6096020e79835f7dbe565bf0a to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0838_1998050100_123_967_12184482.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0838_1998050100_123_967_12184482.zip
ic_ancil_file: ic_ancil_12184482
ifsdata_file: ifsdata_12184482
climate_data_file: clim_data_12184482
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc-client/slots/2/ic_ancil_12184482.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12184482.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_40e13979e0807ae11a67a0469dca8e8f to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12184482.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12184482.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12184482.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12184482.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12184482.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12184482.xml
-- Model is restarting --
Adjusting last_iter, 1640, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12184482.xml
last_cpu_time: 43841
upload_file_number: 68
last_iter: 1631
last_upload: 5875200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034
The child process has been launched with process id: 1686685
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1686685:1686685] [20230115:160049:1673794849:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:01:45 STEP 1632 H=1632:00 +CPU= 47.505
  16:02:12 STEP 1633 H=1633:00 +CPU= 25.770
  16:02:42 STEP 1634 H=1634:00 +CPU= 27.650
  16:03:22 STEP 1635 H=1635:00 +CPU= 38.600
  16:03:50 STEP 1636 H=1636:00 +CPU= 26.258
  16:04:20 STEP 1637 H=1637:00 +CPU= 26.756
  16:05:01 STEP 1638 H=1638:00 +CPU= 39.588
  16:05:30 STEP 1639 H=1639:00 +CPU= 26.852
  16:05:59 STEP 1640 H=1640:00 +CPU= 27.222
  16:06:40 STEP 1641 H=1641:00 +CPU= 39.058
  16:07:10 STEP 1642 H=1642:00 +CPU= 27.565
  16:07:37 STEP 1643 H=1643:00 +CPU= 25.794
  16:08:29 STEP 1644 H=1644:00 +CPU= 44.666
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001644
  16:08:56 STEP 1645 H=1645:00 +CPU= 25.609
  16:09:24 STEP 1646 H=1646:00 +CPU= 26.746
  16:10:09 STEP 1647 H=1647:00 +CPU= 40.482
  16:10:38 STEP 1648 H=1648:00 +CPU= 26.634
  16:11:09 STEP 1649 H=1649:00 +CPU= 27.953
  16:11:53 STEP 1650 H=1650:00 +CPU= 41.084
  16:12:24 STEP 1651 H=1651:00 +CPU= 26.345
  16:12:57 STEP 1652 H=1652:00 +CPU= 27.951
  16:13:41 STEP 1653 H=1653:00 +CPU= 40.145
  16:14:14 STEP 1654 H=1654:00 +CPU= 28.188
  16:14:46 STEP 1655 H=1655:00 +CPU= 28.078
  16:15:38 STEP 1656 H=1656:00 +CPU= 47.260
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_68.zip
Uploading the intermediate file: upload_file_68.zip
  16:16:06 STEP 1657 H=1657:00 +CPU= 25.892
  16:16:33 STEP 1658 H=1658:00 +CPU= 26.134
  16:17:14 STEP 1659 H=1659:00 +CPU= 39.366
  16:17:42 STEP 1660 H=1660:00 +CPU= 26.918
  16:18:09 STEP 1661 H=1661:00 +CPU= 25.787
  16:18:53 STEP 1662 H=1662:00 +CPU= 41.159
  16:19:19 STEP 1663 H=1663:00 +CPU= 25.406
  16:19:47 STEP 1664 H=1664:00 +CPU= 26.762
  16:20:28 STEP 1665 H=1665:00 +CPU= 38.846
  16:20:55 STEP 1666 H=1666:00 +CPU= 26.019
  16:21:26 STEP 1667 H=1667:00 +CPU= 28.086
  16:22:14 STEP 1668 H=1668:00 +CPU= 43.843
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001668
  16:22:41 STEP 1669 H=1669:00 +CPU= 25.576
  16:23:13 STEP 1670 H=1670:00 +CPU= 28.617
  16:23:54 STEP 1671 H=1671:00 +CPU= 38.320
  16:24:21 STEP 1672 H=1672:00 +CPU= 26.098
  16:24:50 STEP 1673 H=1673:00 +CPU= 26.864
  16:25:31 STEP 1674 H=1674:00 +CPU= 39.543
  16:26:00 STEP 1675 H=1675:00 +CPU= 26.097
  16:26:31 STEP 1676 H=1676:00 +CPU= 27.922
  16:27:14 STEP 1677 H=1677:00 +CPU= 39.234
  16:27:43 STEP 1678 H=1678:00 +CPU= 26.671
  16:28:13 STEP 1679 H=1679:00 +CPU= 28.188
  16:29:05 STEP 1680 H=1680:00 +CPU= 46.566
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_69.zip
Uploading the intermediate file: upload_file_69.zip
  16:29:37 STEP 1681 H=1681:00 +CPU= 26.756
  16:30:07 STEP 1682 H=1682:00 +CPU= 27.670
  16:30:52 STEP 1683 H=1683:00 +CPU= 39.795
  16:31:21 STEP 1684 H=1684:00 +CPU= 27.181
  16:31:49 STEP 1685 H=1685:00 +CPU= 25.851
  16:32:35 STEP 1686 H=1686:00 +CPU= 41.658
  16:33:02 STEP 1687 H=1687:00 +CPU= 25.848
  16:33:31 STEP 1688 H=1688:00 +CPU= 26.974
  16:34:15 STEP 1689 H=1689:00 +CPU= 40.602
  16:34:42 STEP 1690 H=1690:00 +CPU= 26.194
  16:35:10 STEP 1691 H=1691:00 +CPU= 26.173
  16:35:57 STEP 1692 H=1692:00 +CPU= 44.264
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001692
  16:36:26 STEP 1693 H=1693:00 +CPU= 26.295
  16:36:55 STEP 1694 H=1694:00 +CPU= 27.342
  16:37:36 STEP 1695 H=1695:00 +CPU= 38.993
  16:38:05 STEP 1696 H=1696:00 +CPU= 27.451
  16:38:33 STEP 1697 H=1697:00 +CPU= 26.027
  16:39:21 STEP 1698 H=1698:00 +CPU= 40.683
  16:39:50 STEP 1699 H=1699:00 +CPU= 25.695
  16:40:19 STEP 1700 H=1700:00 +CPU= 26.675
  16:41:06 STEP 1701 H=1701:00 +CPU= 40.210
  16:41:36 STEP 1702 H=1702:00 +CPU= 26.710
  16:42:06 STEP 1703 H=1703:00 +CPU= 27.222
  16:43:00 STEP 1704 H=1704:00 +CPU= 47.207
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_70.zip
Uploading the intermediate file: upload_file_70.zip
  16:43:30 STEP 1705 H=1705:00 +CPU= 25.985
  16:44:04 STEP 1706 H=1706:00 +CPU= 27.912
  16:44:46 STEP 1707 H=1707:00 +CPU= 39.577
  16:45:14 STEP 1708 H=1708:00 +CPU= 26.251
  16:45:46 STEP 1709 H=1709:00 +CPU= 27.357
  16:46:29 STEP 1710 H=1710:00 +CPU= 39.948
  16:46:58 STEP 1711 H=1711:00 +CPU= 26.345
  16:47:28 STEP 1712 H=1712:00 +CPU= 27.577
  16:48:11 STEP 1713 H=1713:00 +CPU= 39.312
  16:48:43 STEP 1714 H=1714:00 +CPU= 26.623
  16:49:11 STEP 1715 H=1715:00 +CPU= 25.993
  16:50:01 STEP 1716 H=1716:00 +CPU= 43.841
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001716
  16:50:30 STEP 1717 H=1717:00 +CPU= 25.876
  16:50:58 STEP 1718 H=1718:00 +CPU= 26.577
  16:51:44 STEP 1719 H=1719:00 +CPU= 39.771
  16:52:14 STEP 1720 H=1720:00 +CPU= 26.505
  16:52:44 STEP 1721 H=1721:00 +CPU= 27.206
  16:53:28 STEP 1722 H=1722:00 +CPU= 40.180
  16:53:55 STEP 1723 H=1723:00 +CPU= 25.815
  16:54:26 STEP 1724 H=1724:00 +CPU= 28.674
  16:55:09 STEP 1725 H=1725:00 +CPU= 39.227
  16:55:40 STEP 1726 H=1726:00 +CPU= 26.595
  16:56:11 STEP 1727 H=1727:00 +CPU= 27.579
  16:57:03 STEP 1728 H=1728:00 +CPU= 46.632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_71.zip
Uploading the intermediate file: upload_file_71.zip
  16:57:33 STEP 1729 H=1729:00 +CPU= 26.158
  16:58:02 STEP 1730 H=1730:00 +CPU= 26.344
  16:58:48 STEP 1731 H=1731:00 +CPU= 39.690
  16:59:16 STEP 1732 H=1732:00 +CPU= 26.279
  16:59:45 STEP 1733 H=1733:00 +CPU= 26.535
  17:00:30 STEP 1734 H=1734:00 +CPU= 40.540
  17:00:58 STEP 1735 H=1735:00 +CPU= 25.916
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0838
(argv4) batchid: 967
(argv5) wuid: 12184482
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a80facd6096020e79835f7dbe565bf0a to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0838_1998050100_123_967_12184482.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0838_1998050100_123_967_12184482.zip
ic_ancil_file: ic_ancil_12184482
ifsdata_file: ifsdata_12184482
climate_data_file: clim_data_12184482
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc-client/slots/2/ic_ancil_12184482.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12184482.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_40e13979e0807ae11a67a0469dca8e8f to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12184482.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12184482.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12184482.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12184482.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12184482.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12184482.xml
-- Model is restarting --
Adjusting last_iter, 1736, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12184482.xml
last_cpu_time: 47157
upload_file_number: 72
last_iter: 1727
last_upload: 6220800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034
The child process has been launched with process id: 1696619
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1696619:1696619] [20230115:170245:1673798565:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:03:40 STEP 1728 H=1728:00 +CPU= 46.132
  17:04:07 STEP 1729 H=1729:00 +CPU= 25.771
  17:04:38 STEP 1730 H=1730:00 +CPU= 27.308
  17:05:24 STEP 1731 H=1731:00 +CPU= 40.475
  17:05:56 STEP 1732 H=1732:00 +CPU= 27.174
  17:06:23 STEP 1733 H=1733:00 +CPU= 25.883
  17:07:08 STEP 1734 H=1734:00 +CPU= 40.997
  17:07:37 STEP 1735 H=1735:00 +CPU= 25.916
  17:08:07 STEP 1736 H=1736:00 +CPU= 26.575
  17:08:53 STEP 1737 H=1737:00 +CPU= 39.769
  17:09:21 STEP 1738 H=1738:00 +CPU= 26.516
  17:09:49 STEP 1739 H=1739:00 +CPU= 25.765
  17:10:40 STEP 1740 H=1740:00 +CPU= 45.285
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001740
  17:11:09 STEP 1741 H=1741:00 +CPU= 25.849
  17:11:38 STEP 1742 H=1742:00 +CPU= 27.177
  17:12:21 STEP 1743 H=1743:00 +CPU= 39.156
  17:12:54 STEP 1744 H=1744:00 +CPU= 27.535
  17:13:23 STEP 1745 H=1745:00 +CPU= 25.494
  17:14:08 STEP 1746 H=1746:00 +CPU= 40.560
  17:14:38 STEP 1747 H=1747:00 +CPU= 26.234
  17:15:06 STEP 1748 H=1748:00 +CPU= 25.955
  17:15:51 STEP 1749 H=1749:00 +CPU= 39.954
  17:16:18 STEP 1750 H=1750:00 +CPU= 25.831
  17:16:47 STEP 1751 H=1751:00 +CPU= 26.253
  17:17:41 STEP 1752 H=1752:00 +CPU= 47.147
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_72.zip
Uploading the intermediate file: upload_file_72.zip
  17:18:08 STEP 1753 H=1753:00 +CPU= 25.368
  17:18:37 STEP 1754 H=1754:00 +CPU= 26.824
  17:19:17 STEP 1755 H=1755:00 +CPU= 38.683
  17:19:45 STEP 1756 H=1756:00 +CPU= 27.419
  17:20:13 STEP 1757 H=1757:00 +CPU= 25.730
  17:20:56 STEP 1758 H=1758:00 +CPU= 40.041
  17:21:25 STEP 1759 H=1759:00 +CPU= 26.809
  17:21:52 STEP 1760 H=1760:00 +CPU= 26.178
  17:22:33 STEP 1761 H=1761:00 +CPU= 39.813
  17:23:01 STEP 1762 H=1762:00 +CPU= 26.575
  17:23:27 STEP 1763 H=1763:00 +CPU= 25.148
  17:24:11 STEP 1764 H=1764:00 +CPU= 43.337
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001764
  17:24:38 STEP 1765 H=1765:00 +CPU= 25.810
  17:25:05 STEP 1766 H=1766:00 +CPU= 26.518
  17:25:46 STEP 1767 H=1767:00 +CPU= 38.931
  17:26:15 STEP 1768 H=1768:00 +CPU= 27.782
  17:26:41 STEP 1769 H=1769:00 +CPU= 25.122
  17:27:23 STEP 1770 H=1770:00 +CPU= 40.293
  17:27:53 STEP 1771 H=1771:00 +CPU= 27.826
  17:28:21 STEP 1772 H=1772:00 +CPU= 26.200
  17:29:01 STEP 1773 H=1773:00 +CPU= 38.930
  17:29:30 STEP 1774 H=1774:00 +CPU= 27.228
  17:29:57 STEP 1775 H=1775:00 +CPU= 26.169
  17:30:48 STEP 1776 H=1776:00 +CPU= 47.289
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_73.zip
Uploading the intermediate file: upload_file_73.zip
  17:31:14 STEP 1777 H=1777:00 +CPU= 25.269
  17:31:41 STEP 1778 H=1778:00 +CPU= 26.197
  17:32:22 STEP 1779 H=1779:00 +CPU= 38.996
  17:32:49 STEP 1780 H=1780:00 +CPU= 25.976
  17:33:18 STEP 1781 H=1781:00 +CPU= 27.414
  17:34:01 STEP 1782 H=1782:00 +CPU= 39.502
  17:34:27 STEP 1783 H=1783:00 +CPU= 25.485
  17:34:56 STEP 1784 H=1784:00 +CPU= 27.396
  17:35:37 STEP 1785 H=1785:00 +CPU= 39.837
  17:36:05 STEP 1786 H=1786:00 +CPU= 26.397
  17:36:31 STEP 1787 H=1787:00 +CPU= 25.340
  17:37:18 STEP 1788 H=1788:00 +CPU= 45.532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001788
  17:37:48 STEP 1789 H=1789:00 +CPU= 28.233
  17:38:19 STEP 1790 H=1790:00 +CPU= 27.436
  17:39:02 STEP 1791 H=1791:00 +CPU= 39.976
  17:39:32 STEP 1792 H=1792:00 +CPU= 28.152
  17:40:00 STEP 1793 H=1793:00 +CPU= 26.815
  17:40:46 STEP 1794 H=1794:00 +CPU= 42.054
  17:41:13 STEP 1795 H=1795:00 +CPU= 26.101
  17:41:43 STEP 1796 H=1796:00 +CPU= 28.228
  17:42:25 STEP 1797 H=1797:00 +CPU= 40.342
  17:42:52 STEP 1798 H=1798:00 +CPU= 26.534
  17:43:21 STEP 1799 H=1799:00 +CPU= 27.449
  17:44:11 STEP 1800 H=1800:00 +CPU= 46.684
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_74.zip
Uploading the intermediate file: upload_file_74.zip
  17:44:37 STEP 1801 H=1801:00 +CPU= 25.516
  17:45:05 STEP 1802 H=1802:00 +CPU= 26.488
  17:45:45 STEP 1803 H=1803:00 +CPU= 38.432
  17:46:13 STEP 1804 H=1804:00 +CPU= 27.257
  17:46:39 STEP 1805 H=1805:00 +CPU= 25.209
  17:47:21 STEP 1806 H=1806:00 +CPU= 40.012
  17:47:48 STEP 1807 H=1807:00 +CPU= 25.685
  17:48:15 STEP 1808 H=1808:00 +CPU= 26.139
  17:48:57 STEP 1809 H=1809:00 +CPU= 40.248
  17:49:24 STEP 1810 H=1810:00 +CPU= 26.038
  17:49:50 STEP 1811 H=1811:00 +CPU= 25.534
  17:50:36 STEP 1812 H=1812:00 +CPU= 44.384
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001812
  17:51:02 STEP 1813 H=1813:00 +CPU= 25.006
  17:51:31 STEP 1814 H=1814:00 +CPU= 27.500
  17:52:12 STEP 1815 H=1815:00 +CPU= 39.343
  17:52:39 STEP 1816 H=1816:00 +CPU= 26.350
  17:53:08 STEP 1817 H=1817:00 +CPU= 26.943
  17:53:50 STEP 1818 H=1818:00 +CPU= 40.508
  17:54:17 STEP 1819 H=1819:00 +CPU= 25.756
  17:54:45 STEP 1820 H=1820:00 +CPU= 26.388
  17:55:25 STEP 1821 H=1821:00 +CPU= 38.780
  17:55:52 STEP 1822 H=1822:00 +CPU= 26.422
  17:56:20 STEP 1823 H=1823:00 +CPU= 26.631
  17:57:09 STEP 1824 H=1824:00 +CPU= 47.142
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_75.zip
Uploading the intermediate file: upload_file_75.zip
  17:57:36 STEP 1825 H=1825:00 +CPU= 25.971
  17:58:05 STEP 1826 H=1826:00 +CPU= 28.071
  17:58:47 STEP 1827 H=1827:00 +CPU= 38.717
  17:59:14 STEP 1828 H=1828:00 +CPU= 26.445
  17:59:42 STEP 1829 H=1829:00 +CPU= 27.360
  18:00:23 STEP 1830 H=1830:00 +CPU= 39.432
  18:00:50 STEP 1831 H=1831:00 +CPU= 25.676
  18:01:19 STEP 1832 H=1832:00 +CPU= 27.413
  18:01:59 STEP 1833 H=1833:00 +CPU= 38.375
  18:02:28 STEP 1834 H=1834:00 +CPU= 27.746
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1998050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0838
(argv4) batchid: 967
(argv5) wuid: 12184482
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a80facd6096020e79835f7dbe565bf0a to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0838_1998050100_123_967_12184482.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0838_1998050100_123_967_12184482.zip
ic_ancil_file: ic_ancil_12184482
ifsdata_file: ifsdata_12184482
climate_data_file: clim_data_12184482
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b3b72de1fea5937ea6fedf6e21b4e9ba to: /var/lib/boinc-client/slots/2/ic_ancil_12184482.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12184482.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_40e13979e0807ae11a67a0469dca8e8f to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12184482.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12184482.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12184482.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12184482.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12184482.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12184482.xml
-- Model is restarting --
Adjusting last_iter, 1835, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12184482.xml
last_cpu_time: 50532
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034
The child process has been launched with process id: 1706819
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pyroxeen:1:1:1706819:1706819] [20230115:180516:1673802316:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:06:08 STEP 1824 H=1824:00 +CPU= 44.872
  18:06:33 STEP 1825 H=1825:00 +CPU= 24.926
  18:07:04 STEP 1826 H=1826:00 +CPU= 27.591
  18:07:43 STEP 1827 H=1827:00 +CPU= 38.082
  18:08:10 STEP 1828 H=1828:00 +CPU= 26.079
  18:08:39 STEP 1829 H=1829:00 +CPU= 27.336
  18:09:20 STEP 1830 H=1830:00 +CPU= 39.536
  18:09:46 STEP 1831 H=1831:00 +CPU= 25.228
  18:10:18 STEP 1832 H=1832:00 +CPU= 28.284
  18:10:58 STEP 1833 H=1833:00 +CPU= 38.854
  18:11:27 STEP 1834 H=1834:00 +CPU= 26.681
  18:11:55 STEP 1835 H=1835:00 +CPU= 26.604
  18:12:40 STEP 1836 H=1836:00 +CPU= 43.424
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001836
  18:13:10 STEP 1837 H=1837:00 +CPU= 26.961
  18:13:39 STEP 1838 H=1838:00 +CPU= 27.201
  18:14:22 STEP 1839 H=1839:00 +CPU= 40.835
  18:14:52 STEP 1840 H=1840:00 +CPU= 27.324
  18:15:18 STEP 1841 H=1841:00 +CPU= 25.471
  18:16:02 STEP 1842 H=1842:00 +CPU= 41.080
  18:16:28 STEP 1843 H=1843:00 +CPU= 25.054
  18:16:56 STEP 1844 H=1844:00 +CPU= 26.725
  18:17:40 STEP 1845 H=1845:00 +CPU= 41.106
  18:18:07 STEP 1846 H=1846:00 +CPU= 25.811
  18:18:34 STEP 1847 H=1847:00 +CPU= 26.617
  18:19:25 STEP 1848 H=1848:00 +CPU= 47.908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184482/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0838_1998050100_123_967_12184482_0_r1929273034_76.zip
Uploading the intermediate file: upload_file_76.zip
  18:19:53 STEP 1849 H=1849:00 +CPU= 25.104
  18:20:20 STEP 1850 H=1850:00 +CPU= 26.064
  18:21:02 STEP 1851 H=1851:00 +CPU= 40.252
  18:21:31 STEP 1852 H=1852:00 +CPU= 26.603
  18:21:57 STEP 1853 H=1853:00 +CPU= 25.729
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:20:31 1500195 22269739 oifs_43r3_ps_0838_1998050100_123_967_12184482_0 3,456,000 22,437 0.0065
09 May 2023 16:20:23 1500195 22269739 oifs_43r3_ps_0838_1998050100_123_967_12184482_0 2,592,000 16,891 0.0065
09 May 2023 16:20:19 1500195 22269739 oifs_43r3_ps_0838_1998050100_123_967_12184482_0 1,728,000 11,256 0.0065
09 May 2023 16:20:11 1500195 22269739 oifs_43r3_ps_0838_1998050100_123_967_12184482_0 864,000 5,642 0.0065


©2024 climateprediction.net