climateprediction.net home page
Task 22444713

Task 22444713

Name oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0
Workunit 12291115
Created 13 Jun 2024, 0:04:37 UTC
Sent 13 Jun 2024, 17:58:57 UTC
Report deadline 12 Aug 2024, 17:58:57 UTC
Received 15 Jun 2024, 3:55:07 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537687
Run time 17 hours 26 min 55 sec
CPU time 17 hours 26 min 55 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 7.20 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.42 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,377.53 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:873587:873587] [20240615:001725:1718410645:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:873587:873587] [20240615:001725:1718410645:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:873587:873587] [20240615:001725:1718410645:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:873587:873587] [20240615:001725:1718410645:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:17:52 STEP 1536 H= 384:00 +CPU= 23.534
  00:18:17 STEP 1537 H= 384:15 +CPU= 25.317
  00:18:42 STEP 1538 H= 384:30 +CPU= 24.689
  00:19:08 STEP 1539 H= 384:45 +CPU= 25.214
  00:19:36 STEP 1540 H= 385:00 +CPU= 28.071
  00:20:04 STEP 1541 H= 385:15 +CPU= 28.448
  00:20:34 STEP 1542 H= 385:30 +CPU= 29.093
  00:21:03 STEP 1543 H= 385:45 +CPU= 28.807
  00:21:32 STEP 1544 H= 386:00 +CPU= 29.264
  00:21:59 STEP 1545 H= 386:15 +CPU= 26.782
  00:22:20 STEP 1546 H= 386:30 +CPU= 20.238
  00:22:41 STEP 1547 H= 386:45 +CPU= 21.653
  00:23:11 STEP 1548 H= 387:00 +CPU= 29.853
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001548
  00:23:37 STEP 1549 H= 387:15 +CPU= 25.385
  00:24:02 STEP 1550 H= 387:30 +CPU= 25.683
  00:24:30 STEP 1551 H= 387:45 +CPU= 27.309
  00:24:56 STEP 1552 H= 388:00 +CPU= 25.164
  00:25:25 STEP 1553 H= 388:15 +CPU= 28.610
  00:25:55 STEP 1554 H= 388:30 +CPU= 29.390
  00:26:25 STEP 1555 H= 388:45 +CPU= 29.241
  00:26:55 STEP 1556 H= 389:00 +CPU= 29.940
  00:27:26 STEP 1557 H= 389:15 +CPU= 29.243
  00:27:55 STEP 1558 H= 389:30 +CPU= 28.694
  00:28:24 STEP 1559 H= 389:45 +CPU= 28.159
  00:29:00 STEP 1560 H= 390:00 +CPU= 34.881
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001560
  00:29:28 STEP 1561 H= 390:15 +CPU= 28.575
  00:29:58 STEP 1562 H= 390:30 +CPU= 29.150
  00:30:27 STEP 1563 H= 390:45 +CPU= 28.805
  00:30:57 STEP 1564 H= 391:00 +CPU= 29.674
  00:31:26 STEP 1565 H= 391:15 +CPU= 28.700
  00:31:55 STEP 1566 H= 391:30 +CPU= 29.027
  00:32:24 STEP 1567 H= 391:45 +CPU= 29.037
  00:32:54 STEP 1568 H= 392:00 +CPU= 30.102
  00:33:24 STEP 1569 H= 392:15 +CPU= 28.616
  00:33:50 STEP 1570 H= 392:30 +CPU= 25.616
  00:34:14 STEP 1571 H= 392:45 +CPU= 24.811
  00:34:46 STEP 1572 H= 393:00 +CPU= 30.969
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001572
  00:35:12 STEP 1573 H= 393:15 +CPU= 26.520
  00:35:40 STEP 1574 H= 393:30 +CPU= 26.357
  00:36:07 STEP 1575 H= 393:45 +CPU= 27.017
  00:36:33 STEP 1576 H= 394:00 +CPU= 25.505
  00:37:02 STEP 1577 H= 394:15 +CPU= 28.722
  00:37:33 STEP 1578 H= 394:30 +CPU= 29.364
  00:38:03 STEP 1579 H= 394:45 +CPU= 29.340
  00:38:34 STEP 1580 H= 395:00 +CPU= 29.666
  00:39:03 STEP 1581 H= 395:15 +CPU= 28.126
  00:39:24 STEP 1582 H= 395:30 +CPU= 21.252
  00:39:46 STEP 1583 H= 395:45 +CPU= 21.720
  00:40:18 STEP 1584 H= 396:00 +CPU= 31.946
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001584
  00:40:43 STEP 1585 H= 396:15 +CPU= 24.669
  00:41:08 STEP 1586 H= 396:30 +CPU= 25.471
  00:41:34 STEP 1587 H= 396:45 +CPU= 25.479
  00:41:58 STEP 1588 H= 397:00 +CPU= 24.477
  00:42:27 STEP 1589 H= 397:15 +CPU= 27.937
  00:42:56 STEP 1590 H= 397:30 +CPU= 29.445
  00:43:26 STEP 1591 H= 397:45 +CPU= 29.256
  00:43:56 STEP 1592 H= 398:00 +CPU= 29.752
  00:44:25 STEP 1593 H= 398:15 +CPU= 29.205
  00:44:54 STEP 1594 H= 398:30 +CPU= 29.346
  00:45:18 STEP 1595 H= 398:45 +CPU= 23.818
  00:45:44 STEP 1596 H= 399:00 +CPU= 26.369
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001596
  00:46:09 STEP 1597 H= 399:15 +CPU= 24.230
  00:46:33 STEP 1598 H= 399:30 +CPU= 23.969
  00:46:57 STEP 1599 H= 399:45 +CPU= 24.129
  00:47:23 STEP 1600 H= 400:00 +CPU= 25.515
  00:47:49 STEP 1601 H= 400:15 +CPU= 25.757
  00:48:16 STEP 1602 H= 400:30 +CPU= 26.952
  00:48:45 STEP 1603 H= 400:45 +CPU= 29.102
  00:49:15 STEP 1604 H= 401:00 +CPU= 29.255
  00:49:46 STEP 1605 H= 401:15 +CPU= 29.701
  00:50:15 STEP 1606 H= 401:30 +CPU= 28.819
  00:50:44 STEP 1607 H= 401:45 +CPU= 29.040
  00:51:21 STEP 1608 H= 402:00 +CPU= 36.160
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001608
  00:51:50 STEP 1609 H= 402:15 +CPU= 29.494
  00:52:20 STEP 1610 H= 402:30 +CPU= 28.969
  00:52:48 STEP 1611 H= 402:45 +CPU= 28.525
  00:53:18 STEP 1612 H= 403:00 +CPU= 29.311
  00:53:48 STEP 1613 H= 403:15 +CPU= 29.316
  00:54:17 STEP 1614 H= 403:30 +CPU= 29.391
  00:54:46 STEP 1615 H= 403:45 +CPU= 28.744
  00:55:15 STEP 1616 H= 404:00 +CPU= 29.210
  00:55:46 STEP 1617 H= 404:15 +CPU= 29.103
  00:56:15 STEP 1618 H= 404:30 +CPU= 29.519
  00:56:38 STEP 1619 H= 404:45 +CPU= 23.018
  00:57:04 STEP 1620 H= 405:00 +CPU= 25.984
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001620
  00:57:26 STEP 1621 H= 405:15 +CPU= 21.247
  00:57:50 STEP 1622 H= 405:30 +CPU= 24.909
  00:58:14 STEP 1623 H= 405:45 +CPU= 23.341
  00:58:41 STEP 1624 H= 406:00 +CPU= 27.313
  00:59:05 STEP 1625 H= 406:15 +CPU= 23.411
  00:59:32 STEP 1626 H= 406:30 +CPU= 26.584
  01:00:00 STEP 1627 H= 406:45 +CPU= 28.322
  01:00:30 STEP 1628 H= 407:00 +CPU= 29.994
  01:01:00 STEP 1629 H= 407:15 +CPU= 29.192
  01:01:29 STEP 1630 H= 407:30 +CPU= 29.103
  01:01:58 STEP 1631 H= 407:45 +CPU= 29.106
  01:02:38 STEP 1632 H= 408:00 +CPU= 39.789
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0_r287661234_16.zip
Uploading the intermediate file: upload_file_16.zip
  01:03:09 STEP 1633 H= 408:15 +CPU= 28.473
  01:03:38 STEP 1634 H= 408:30 +CPU= 28.043
  01:04:07 STEP 1635 H= 408:45 +CPU= 27.839
  01:04:37 STEP 1636 H= 409:00 +CPU= 29.978
  01:05:06 STEP 1637 H= 409:15 +CPU= 28.874
  01:05:36 STEP 1638 H= 409:30 +CPU= 28.169
  01:06:05 STEP 1639 H= 409:45 +CPU= 29.202
  01:06:36 STEP 1640 H= 410:00 +CPU= 30.164
  01:07:05 STEP 1641 H= 410:15 +CPU= 29.104
  01:07:35 STEP 1642 H= 410:30 +CPU= 29.393
  01:07:58 STEP 1643 H= 410:45 +CPU= 23.716
  01:08:24 STEP 1644 H= 411:00 +CPU= 25.429
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001644
  01:08:44 STEP 1645 H= 411:15 +CPU= 20.355
  01:09:11 STEP 1646 H= 411:30 +CPU= 26.346
  01:09:36 STEP 1647 H= 411:45 +CPU= 25.029
  01:10:01 STEP 1648 H= 412:00 +CPU= 24.954
  01:10:27 STEP 1649 H= 412:15 +CPU= 26.099
  01:10:53 STEP 1650 H= 412:30 +CPU= 25.711
  01:11:22 STEP 1651 H= 412:45 +CPU= 28.485
  01:11:51 STEP 1652 H= 413:00 +CPU= 29.285
  01:12:20 STEP 1653 H= 413:15 +CPU= 28.818
  01:12:49 STEP 1654 H= 413:30 +CPU= 28.714
  01:13:19 STEP 1655 H= 413:45 +CPU= 29.456
  01:13:50 STEP 1656 H= 414:00 +CPU= 31.342
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001656
  01:14:15 STEP 1657 H= 414:15 +CPU= 24.687
  01:14:39 STEP 1658 H= 414:30 +CPU= 24.056
  01:15:04 STEP 1659 H= 414:45 +CPU= 24.804
  01:15:31 STEP 1660 H= 415:00 +CPU= 26.255
  01:15:58 STEP 1661 H= 415:15 +CPU= 26.928
  01:16:20 STEP 1662 H= 415:30 +CPU= 22.745
  01:16:47 STEP 1663 H= 415:45 +CPU= 26.873
  01:17:17 STEP 1664 H= 416:00 +CPU= 29.351
  01:17:46 STEP 1665 H= 416:15 +CPU= 28.761
  01:18:15 STEP 1666 H= 416:30 +CPU= 29.300
  01:18:44 STEP 1667 H= 416:45 +CPU= 28.778
  01:19:20 STEP 1668 H= 417:00 +CPU= 36.050
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001668
  01:19:48 STEP 1669 H= 417:15 +CPU= 27.338
  01:20:14 STEP 1670 H= 417:30 +CPU= 25.542
  01:20:39 STEP 1671 H= 417:45 +CPU= 25.668
  01:21:05 STEP 1672 H= 418:00 +CPU= 25.635
  01:21:30 STEP 1673 H= 418:15 +CPU= 24.762
  01:21:54 STEP 1674 H= 418:30 +CPU= 24.117
  01:22:19 STEP 1675 H= 418:45 +CPU= 24.880
  01:22:45 STEP 1676 H= 419:00 +CPU= 25.788
  01:23:16 STEP 1677 H= 419:15 +CPU= 29.233
  01:23:45 STEP 1678 H= 419:30 +CPU= 29.071
  01:24:15 STEP 1679 H= 419:45 +CPU= 29.233
  01:24:55 STEP 1680 H= 420:00 +CPU= 38.420
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001680
  01:25:25 STEP 1681 H= 420:15 +CPU= 28.437
  01:25:48 STEP 1682 H= 420:30 +CPU= 23.646
  01:26:12 STEP 1683 H= 420:45 +CPU= 23.943
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a25g
(argv4) batchid: 1018
(argv5) wuid: 12291115
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e31affa81de101049f25e5f9a910b456 to: /var/lib/boinc-client/slots/20/oifs_43r3_bl_a25g_2016092300_20_1018_12291115.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_bl_a25g_2016092300_20_1018_12291115.zip
ic_ancil_file: ic_ancil_12291115
ifsdata_file: ifsdata_12291115
climate_data_file: clim_data_12291115
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/20/ic_ancil_12291115.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12291115.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_6dc255f0d3d1e81652d13350879cff66 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12291115.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12291115.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12291115.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12291115.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12291115.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12291115.xml
-- Model is restarting --
Adjusting last_iter, 1684, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12291115.xml
last_cpu_time: 56188
upload_file_number: 17
last_iter: 1681
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0_r287661234
The child process has been launched with process id: 876316
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:876316:876316] [20240615:012725:1718414845:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:27:56 STEP 1680 H= 420:00 +CPU= 27.414
  01:28:22 STEP 1681 H= 420:15 +CPU= 26.020
  01:28:51 STEP 1682 H= 420:30 +CPU= 28.187
  01:29:20 STEP 1683 H= 420:45 +CPU= 29.101
  01:29:50 STEP 1684 H= 421:00 +CPU= 29.491
  01:30:19 STEP 1685 H= 421:15 +CPU= 28.475
  01:30:48 STEP 1686 H= 421:30 +CPU= 28.921
  01:31:18 STEP 1687 H= 421:45 +CPU= 28.891
  01:31:46 STEP 1688 H= 422:00 +CPU= 28.490
  01:32:15 STEP 1689 H= 422:15 +CPU= 28.831
  01:32:44 STEP 1690 H= 422:30 +CPU= 28.750
  01:33:13 STEP 1691 H= 422:45 +CPU= 28.990
  01:33:49 STEP 1692 H= 423:00 +CPU= 35.415
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001692
  01:34:18 STEP 1693 H= 423:15 +CPU= 29.184
  01:34:47 STEP 1694 H= 423:30 +CPU= 28.457
  01:35:16 STEP 1695 H= 423:45 +CPU= 28.736
  01:35:47 STEP 1696 H= 424:00 +CPU= 29.518
  01:36:17 STEP 1697 H= 424:15 +CPU= 29.197
  01:36:46 STEP 1698 H= 424:30 +CPU= 28.686
  01:37:14 STEP 1699 H= 424:45 +CPU= 27.273
  01:37:37 STEP 1700 H= 425:00 +CPU= 22.671
  01:37:58 STEP 1701 H= 425:15 +CPU= 21.463
  01:38:24 STEP 1702 H= 425:30 +CPU= 25.066
  01:38:49 STEP 1703 H= 425:45 +CPU= 24.970
  01:39:20 STEP 1704 H= 426:00 +CPU= 30.935
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001704
  01:39:45 STEP 1705 H= 426:15 +CPU= 25.471
  01:40:11 STEP 1706 H= 426:30 +CPU= 25.393
  01:40:39 STEP 1707 H= 426:45 +CPU= 27.952
  01:41:09 STEP 1708 H= 427:00 +CPU= 29.651
  01:41:39 STEP 1709 H= 427:15 +CPU= 28.976
  01:42:09 STEP 1710 H= 427:30 +CPU= 29.458
  01:42:38 STEP 1711 H= 427:45 +CPU= 28.636
  01:43:08 STEP 1712 H= 428:00 +CPU= 29.036
  01:43:34 STEP 1713 H= 428:15 +CPU= 26.409
  01:43:56 STEP 1714 H= 428:30 +CPU= 21.443
  01:44:17 STEP 1715 H= 428:45 +CPU= 21.082
  01:44:46 STEP 1716 H= 429:00 +CPU= 29.326
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001716
  01:45:11 STEP 1717 H= 429:15 +CPU= 24.351
  01:45:35 STEP 1718 H= 429:30 +CPU= 24.479
  01:45:59 STEP 1719 H= 429:45 +CPU= 24.100
  01:46:25 STEP 1720 H= 430:00 +CPU= 25.434
  01:46:54 STEP 1721 H= 430:15 +CPU= 28.482
  01:47:23 STEP 1722 H= 430:30 +CPU= 29.060
  01:47:53 STEP 1723 H= 430:45 +CPU= 29.039
  01:48:23 STEP 1724 H= 431:00 +CPU= 29.336
  01:48:53 STEP 1725 H= 431:15 +CPU= 28.706
  01:49:21 STEP 1726 H= 431:30 +CPU= 27.965
  01:49:41 STEP 1727 H= 431:45 +CPU= 19.720
  01:50:09 STEP 1728 H= 432:00 +CPU= 28.101
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0_r287661234_17.zip
Uploading the intermediate file: upload_file_17.zip
  01:50:31 STEP 1729 H= 432:15 +CPU= 21.901
  01:50:55 STEP 1730 H= 432:30 +CPU= 24.237
  01:51:17 STEP 1731 H= 432:45 +CPU= 22.036
  01:51:39 STEP 1732 H= 433:00 +CPU= 22.243
  01:52:04 STEP 1733 H= 433:15 +CPU= 24.801
  01:52:27 STEP 1734 H= 433:30 +CPU= 22.955
  01:52:54 STEP 1735 H= 433:45 +CPU= 26.669
  01:53:20 STEP 1736 H= 434:00 +CPU= 26.195
  01:53:49 STEP 1737 H= 434:15 +CPU= 28.060
  01:54:19 STEP 1738 H= 434:30 +CPU= 29.017
  01:54:49 STEP 1739 H= 434:45 +CPU= 29.105
  01:55:25 STEP 1740 H= 435:00 +CPU= 35.480
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001740
  01:55:55 STEP 1741 H= 435:15 +CPU= 28.938
  01:56:24 STEP 1742 H= 435:30 +CPU= 28.755
  01:56:53 STEP 1743 H= 435:45 +CPU= 29.049
  01:57:23 STEP 1744 H= 436:00 +CPU= 29.299
  01:57:52 STEP 1745 H= 436:15 +CPU= 29.291
  01:58:23 STEP 1746 H= 436:30 +CPU= 29.413
  01:58:52 STEP 1747 H= 436:45 +CPU= 29.170
  01:59:22 STEP 1748 H= 437:00 +CPU= 29.762
  01:59:51 STEP 1749 H= 437:15 +CPU= 28.760
  02:00:20 STEP 1750 H= 437:30 +CPU= 28.706
  02:00:50 STEP 1751 H= 437:45 +CPU= 29.294
  02:01:26 STEP 1752 H= 438:00 +CPU= 35.421
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001752
  02:01:55 STEP 1753 H= 438:15 +CPU= 28.576
  02:02:24 STEP 1754 H= 438:30 +CPU= 28.959
  02:02:53 STEP 1755 H= 438:45 +CPU= 28.720
  02:03:23 STEP 1756 H= 439:00 +CPU= 29.472
  02:03:53 STEP 1757 H= 439:15 +CPU= 29.420
  02:04:22 STEP 1758 H= 439:30 +CPU= 29.003
  02:04:50 STEP 1759 H= 439:45 +CPU= 27.202
  02:05:11 STEP 1760 H= 440:00 +CPU= 20.872
  02:05:35 STEP 1761 H= 440:15 +CPU= 24.363
  02:06:00 STEP 1762 H= 440:30 +CPU= 24.946
  02:06:25 STEP 1763 H= 440:45 +CPU= 24.279
  02:06:57 STEP 1764 H= 441:00 +CPU= 31.707
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001764
  02:07:23 STEP 1765 H= 441:15 +CPU= 25.842
  02:07:48 STEP 1766 H= 441:30 +CPU= 25.262
  02:08:17 STEP 1767 H= 441:45 +CPU= 28.829
  02:08:48 STEP 1768 H= 442:00 +CPU= 29.786
  02:09:17 STEP 1769 H= 442:15 +CPU= 29.165
  02:09:47 STEP 1770 H= 442:30 +CPU= 28.887
  02:10:16 STEP 1771 H= 442:45 +CPU= 29.084
  02:10:46 STEP 1772 H= 443:00 +CPU= 29.726
  02:11:10 STEP 1773 H= 443:15 +CPU= 23.773
  02:11:34 STEP 1774 H= 443:30 +CPU= 23.959
  02:12:01 STEP 1775 H= 443:45 +CPU= 27.052
  02:12:35 STEP 1776 H= 444:00 +CPU= 34.066
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001776
  02:13:02 STEP 1777 H= 444:15 +CPU= 26.024
  02:13:28 STEP 1778 H= 444:30 +CPU= 26.269
  02:13:54 STEP 1779 H= 444:45 +CPU= 25.349
  02:14:23 STEP 1780 H= 445:00 +CPU= 28.911
  02:14:53 STEP 1781 H= 445:15 +CPU= 28.622
  02:15:22 STEP 1782 H= 445:30 +CPU= 29.194
  02:15:53 STEP 1783 H= 445:45 +CPU= 29.466
  02:16:23 STEP 1784 H= 446:00 +CPU= 29.270
  02:16:52 STEP 1785 H= 446:15 +CPU= 29.281
  02:17:18 STEP 1786 H= 446:30 +CPU= 26.312
  02:17:40 STEP 1787 H= 446:45 +CPU= 21.804
  02:18:11 STEP 1788 H= 447:00 +CPU= 30.823
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001788
  02:18:37 STEP 1789 H= 447:15 +CPU= 26.143
  02:19:03 STEP 1790 H= 447:30 +CPU= 25.837
  02:19:29 STEP 1791 H= 447:45 +CPU= 25.931
  02:19:53 STEP 1792 H= 448:00 +CPU= 23.783
  02:20:20 STEP 1793 H= 448:15 +CPU= 26.815
  02:20:49 STEP 1794 H= 448:30 +CPU= 28.717
  02:21:19 STEP 1795 H= 448:45 +CPU= 28.901
  02:21:49 STEP 1796 H= 449:00 +CPU= 29.609
  02:22:19 STEP 1797 H= 449:15 +CPU= 29.329
  02:22:47 STEP 1798 H= 449:30 +CPU= 28.701
  02:23:16 STEP 1799 H= 449:45 +CPU= 28.492
  02:23:52 STEP 1800 H= 450:00 +CPU= 35.525
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001800
  02:24:21 STEP 1801 H= 450:15 +CPU= 28.967
  02:24:51 STEP 1802 H= 450:30 +CPU= 28.966
  02:25:20 STEP 1803 H= 450:45 +CPU= 29.087
  02:25:50 STEP 1804 H= 451:00 +CPU= 29.741
  02:26:19 STEP 1805 H= 451:15 +CPU= 29.044
  02:26:48 STEP 1806 H= 451:30 +CPU= 28.829
  02:27:17 STEP 1807 H= 451:45 +CPU= 28.755
  02:27:47 STEP 1808 H= 452:00 +CPU= 29.615
  02:28:16 STEP 1809 H= 452:15 +CPU= 28.246
  02:28:46 STEP 1810 H= 452:30 +CPU= 28.988
  02:29:08 STEP 1811 H= 452:45 +CPU= 22.006
  02:29:35 STEP 1812 H= 453:00 +CPU= 27.137
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001812
  02:29:59 STEP 1813 H= 453:15 +CPU= 23.757
  02:30:23 STEP 1814 H= 453:30 +CPU= 23.967
  02:30:47 STEP 1815 H= 453:45 +CPU= 23.913
  02:31:13 STEP 1816 H= 454:00 +CPU= 25.398
  02:31:37 STEP 1817 H= 454:15 +CPU= 24.681
  02:32:05 STEP 1818 H= 454:30 +CPU= 27.429
  02:32:34 STEP 1819 H= 454:45 +CPU= 28.379
  02:33:03 STEP 1820 H= 455:00 +CPU= 29.031
  02:33:33 STEP 1821 H= 455:15 +CPU= 29.245
  02:34:02 STEP 1822 H= 455:30 +CPU= 29.053
  02:34:31 STEP 1823 H= 455:45 +CPU= 28.897
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a25g
(argv4) batchid: 1018
(argv5) wuid: 12291115
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e31affa81de101049f25e5f9a910b456 to: /var/lib/boinc-client/slots/20/oifs_43r3_bl_a25g_2016092300_20_1018_12291115.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_bl_a25g_2016092300_20_1018_12291115.zip
ic_ancil_file: ic_ancil_12291115
ifsdata_file: ifsdata_12291115
climate_data_file: clim_data_12291115
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/20/ic_ancil_12291115.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12291115.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_6dc255f0d3d1e81652d13350879cff66 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12291115.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12291115.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12291115.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12291115.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12291115.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12291115.xml
-- Model is restarting --
Adjusting last_iter, 1824, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12291115.xml
last_cpu_time: 60191
upload_file_number: 18
last_iter: 1825
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0_r287661234
The child process has been launched with process id: 878625
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:box.zenlime.org:1:1:878625:878625] [20240615:023631:1718418991:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0_r287661234_18.zip
Uploading the intermediate file: upload_file_18.zip
  02:37:04 STEP 1824 H= 456:00 +CPU= 28.517
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001824
  02:37:30 STEP 1825 H= 456:15 +CPU= 24.933
  02:37:56 STEP 1826 H= 456:30 +CPU= 25.831
  02:38:24 STEP 1827 H= 456:45 +CPU= 28.206
  02:38:54 STEP 1828 H= 457:00 +CPU= 29.295
  02:39:23 STEP 1829 H= 457:15 +CPU= 28.898
  02:39:53 STEP 1830 H= 457:30 +CPU= 29.465
  02:40:21 STEP 1831 H= 457:45 +CPU= 28.752
  02:40:51 STEP 1832 H= 458:00 +CPU= 29.613
  02:41:20 STEP 1833 H= 458:15 +CPU= 29.130
  02:41:50 STEP 1834 H= 458:30 +CPU= 29.183
  02:42:20 STEP 1835 H= 458:45 +CPU= 29.516
  02:42:57 STEP 1836 H= 459:00 +CPU= 36.061
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001836
  02:43:26 STEP 1837 H= 459:15 +CPU= 29.296
  02:43:55 STEP 1838 H= 459:30 +CPU= 28.933
  02:44:24 STEP 1839 H= 459:45 +CPU= 28.680
  02:44:54 STEP 1840 H= 460:00 +CPU= 29.909
  02:45:23 STEP 1841 H= 460:15 +CPU= 28.887
  02:45:52 STEP 1842 H= 460:30 +CPU= 28.550
  02:46:21 STEP 1843 H= 460:45 +CPU= 27.901
  02:46:45 STEP 1844 H= 461:00 +CPU= 23.831
  02:47:07 STEP 1845 H= 461:15 +CPU= 22.048
  02:47:30 STEP 1846 H= 461:30 +CPU= 23.197
  02:47:55 STEP 1847 H= 461:45 +CPU= 24.757
  02:48:23 STEP 1848 H= 462:00 +CPU= 28.188
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001848
  02:48:47 STEP 1849 H= 462:15 +CPU= 23.500
  02:49:10 STEP 1850 H= 462:30 +CPU= 22.942
  02:49:38 STEP 1851 H= 462:45 +CPU= 27.908
  02:50:08 STEP 1852 H= 463:00 +CPU= 29.634
  02:50:37 STEP 1853 H= 463:15 +CPU= 28.940
  02:51:06 STEP 1854 H= 463:30 +CPU= 28.796
  02:51:35 STEP 1855 H= 463:45 +CPU= 28.751
  02:52:04 STEP 1856 H= 464:00 +CPU= 29.230
  02:52:24 STEP 1857 H= 464:15 +CPU= 19.595
  02:52:43 STEP 1858 H= 464:30 +CPU= 18.911
  02:53:04 STEP 1859 H= 464:45 +CPU= 21.147
  02:53:35 STEP 1860 H= 465:00 +CPU= 30.173
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001860
  02:53:59 STEP 1861 H= 465:15 +CPU= 23.947
  02:54:22 STEP 1862 H= 465:30 +CPU= 23.878
  02:54:48 STEP 1863 H= 465:45 +CPU= 25.700
  02:55:17 STEP 1864 H= 466:00 +CPU= 27.890
  02:55:47 STEP 1865 H= 466:15 +CPU= 28.888
  02:56:17 STEP 1866 H= 466:30 +CPU= 28.182
  02:56:47 STEP 1867 H= 466:45 +CPU= 29.687
  02:57:17 STEP 1868 H= 467:00 +CPU= 29.699
  02:57:47 STEP 1869 H= 467:15 +CPU= 29.176
  02:58:17 STEP 1870 H= 467:30 +CPU= 28.724
  02:58:42 STEP 1871 H= 467:45 +CPU= 24.953
  02:59:12 STEP 1872 H= 468:00 +CPU= 29.329
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001872
  02:59:38 STEP 1873 H= 468:15 +CPU= 26.312
  03:00:04 STEP 1874 H= 468:30 +CPU= 26.290
  03:00:28 STEP 1875 H= 468:45 +CPU= 24.074
  03:00:53 STEP 1876 H= 469:00 +CPU= 24.730
  03:01:18 STEP 1877 H= 469:15 +CPU= 24.348
  03:01:47 STEP 1878 H= 469:30 +CPU= 28.448
  03:02:18 STEP 1879 H= 469:45 +CPU= 29.372
  03:02:48 STEP 1880 H= 470:00 +CPU= 30.042
  03:03:18 STEP 1881 H= 470:15 +CPU= 29.080
  03:03:47 STEP 1882 H= 470:30 +CPU= 29.000
  03:04:16 STEP 1883 H= 470:45 +CPU= 29.034
  03:04:49 STEP 1884 H= 471:00 +CPU= 32.703
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001884
  03:05:13 STEP 1885 H= 471:15 +CPU= 23.794
  03:05:37 STEP 1886 H= 471:30 +CPU= 23.861
  03:06:01 STEP 1887 H= 471:45 +CPU= 24.331
  03:06:28 STEP 1888 H= 472:00 +CPU= 26.771
  03:06:54 STEP 1889 H= 472:15 +CPU= 25.376
  03:07:17 STEP 1890 H= 472:30 +CPU= 23.351
  03:07:45 STEP 1891 H= 472:45 +CPU= 28.195
  03:08:16 STEP 1892 H= 473:00 +CPU= 30.277
  03:08:45 STEP 1893 H= 473:15 +CPU= 29.340
  03:09:14 STEP 1894 H= 473:30 +CPU= 28.576
  03:09:43 STEP 1895 H= 473:45 +CPU= 29.233
  03:10:13 STEP 1896 H= 474:00 +CPU= 29.872
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001896
  03:10:34 STEP 1897 H= 474:15 +CPU= 20.410
  03:10:55 STEP 1898 H= 474:30 +CPU= 21.476
  03:11:19 STEP 1899 H= 474:45 +CPU= 23.442
  03:11:39 STEP 1900 H= 475:00 +CPU= 20.695
  03:12:01 STEP 1901 H= 475:15 +CPU= 21.555
  03:12:26 STEP 1902 H= 475:30 +CPU= 25.104
  03:12:52 STEP 1903 H= 475:45 +CPU= 25.862
  03:13:21 STEP 1904 H= 476:00 +CPU= 29.070
  03:13:50 STEP 1905 H= 476:15 +CPU= 28.654
  03:14:18 STEP 1906 H= 476:30 +CPU= 28.036
  03:14:47 STEP 1907 H= 476:45 +CPU= 28.756
  03:15:24 STEP 1908 H= 477:00 +CPU= 36.199
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001908
  03:15:53 STEP 1909 H= 477:15 +CPU= 28.725
  03:16:22 STEP 1910 H= 477:30 +CPU= 27.269
  03:16:43 STEP 1911 H= 477:45 +CPU= 20.962
  03:17:04 STEP 1912 H= 478:00 +CPU= 20.997
  03:17:26 STEP 1913 H= 478:15 +CPU= 22.211
  03:17:50 STEP 1914 H= 478:30 +CPU= 23.884
  03:18:15 STEP 1915 H= 478:45 +CPU= 25.292
  03:18:41 STEP 1916 H= 479:00 +CPU= 26.202
  03:19:07 STEP 1917 H= 479:15 +CPU= 25.118
  03:19:34 STEP 1918 H= 479:30 +CPU= 27.197
  03:20:04 STEP 1919 H= 479:45 +CPU= 28.635
  03:20:42 STEP 1920 H= 480:00 +CPU= 37.049
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.8639097506E-21      I       0.3316646507E-06      I
ICloud fractionI      0.0000000000E+00      I       0.1769785317E-01      I
I  Relat. Hum. I      0.2046642796E+00      I       0.2635378729E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3283408038E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    325598260 BYTES IN      7 RECORDS TRANSFERRED IN    0.0010 SECONDS325598.3 Mbytes/s, TOTAL TIME=    0.0134( 7.5%)
 03:20:42 STEP 1920 H= 480:00 +CPU= 37.049
 END CNT3
 NSTEP =  1920 CNT0   000000000  26.708  26.708  26.955   0.991               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         2860/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       46      1044068     2860 ************     6.0     1.4     4.6     2.4     9.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 2651084.1 2651084.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 2646932.4 2646932.4     99.84      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    5277.9    5277.9     20.90      0.00
   9 SPCM     - SPECTRAL COMP.                      96     920.8     920.8      3.33      0.00
  10 SCAN2M - PHYSICS                               97   11661.9   11661.9     42.67      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     807.6     807.6      0.24      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      79.1      79.1      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      27.2      27.2      0.01      0.00
  27 RESTART FILES - WRITING                         2    3226.2    3226.2      0.24      0.00
  28 RESTART FILES - READING                         1    3562.7    3562.7      0.13      0.00
  29 SU4FPOS IN CNT4                                 8       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    5634.4    5634.4      1.70      0.00
  31 POSDDH IN STEPO                                 1      18.3      18.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    5503.6    5503.6     20.14      0.00
  39 SU0YOMB                                         1     351.7     351.7      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      28.4      28.4      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     550.3     550.3      1.99      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      71.6      71.6      0.26      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     393.1     393.1      3.14      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     223.9     223.9      1.69      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      73.6      73.6      0.56      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     108.8     108.8      0.87      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      38.4      38.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      65.0      65.0      0.52      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      35.6      35.6      0.27      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212     137.2     137.2      1.10      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      57.8      57.8      0.44      0.00
 400 GSTATS                                      12215       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11311       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     2651.1 CPU TIME    2624.5 VECTOR TIME     2624.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      652.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/20/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/20/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291115/ICMUAh7zg+001848
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0_r287661234_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
03:24:10 (878616): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Jun 2024 03:56:39 1537687 22444713 oifs_43r3_bl_a25g_2016092300_20_1018_12291115_0 1,727,100 62,815 0.0364


©2024 climateprediction.net