climateprediction.net home page
Task 22447665

Task 22447665

Name oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0
Workunit 12294048
Created 13 Jun 2024, 4:17:51 UTC
Sent 13 Jun 2024, 18:12:09 UTC
Report deadline 12 Aug 2024, 18:12:09 UTC
Received 18 Jun 2024, 4:49:46 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1469580
Run time 12 hours 59 min 26 sec
CPU time 12 hours 59 min 26 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.26 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,006.29 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,397.13 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
penIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:24477:24477] [20240618:034630:1718675190:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0_r1209853097_15.zip
Uploading the intermediate file: upload_file_15.zip
  03:47:00 STEP 1536 H= 384:00 +CPU= 25.128
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001536
  03:47:23 STEP 1537 H= 384:15 +CPU= 22.884
  03:47:46 STEP 1538 H= 384:30 +CPU= 23.600
  03:48:10 STEP 1539 H= 384:45 +CPU= 23.311
  03:48:33 STEP 1540 H= 385:00 +CPU= 23.695
  03:48:57 STEP 1541 H= 385:15 +CPU= 23.252
  03:49:20 STEP 1542 H= 385:30 +CPU= 23.378
  03:49:43 STEP 1543 H= 385:45 +CPU= 22.824
  03:50:06 STEP 1544 H= 386:00 +CPU= 22.970
  03:50:30 STEP 1545 H= 386:15 +CPU= 24.364
  03:50:55 STEP 1546 H= 386:30 +CPU= 24.785
  03:51:18 STEP 1547 H= 386:45 +CPU= 22.521
  03:51:46 STEP 1548 H= 387:00 +CPU= 28.131
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001548
  03:52:10 STEP 1549 H= 387:15 +CPU= 24.066
  03:52:35 STEP 1550 H= 387:30 +CPU= 24.878
  03:52:58 STEP 1551 H= 387:45 +CPU= 23.295
  03:53:22 STEP 1552 H= 388:00 +CPU= 23.773
  03:53:47 STEP 1553 H= 388:15 +CPU= 24.597
  03:54:13 STEP 1554 H= 388:30 +CPU= 24.995
  03:54:35 STEP 1555 H= 388:45 +CPU= 22.947
  03:54:59 STEP 1556 H= 389:00 +CPU= 23.243
  03:55:23 STEP 1557 H= 389:15 +CPU= 24.045
  03:55:48 STEP 1558 H= 389:30 +CPU= 25.117
  03:56:10 STEP 1559 H= 389:45 +CPU= 22.144
  03:56:38 STEP 1560 H= 390:00 +CPU= 27.729
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001560
  03:57:02 STEP 1561 H= 390:15 +CPU= 24.221
  03:57:25 STEP 1562 H= 390:30 +CPU= 23.086
  03:57:48 STEP 1563 H= 390:45 +CPU= 22.449
  03:58:11 STEP 1564 H= 391:00 +CPU= 23.187
  03:58:36 STEP 1565 H= 391:15 +CPU= 24.842
  03:59:00 STEP 1566 H= 391:30 +CPU= 24.080
  03:59:22 STEP 1567 H= 391:45 +CPU= 22.278
  03:59:46 STEP 1568 H= 392:00 +CPU= 23.387
  04:00:10 STEP 1569 H= 392:15 +CPU= 24.097
  04:00:34 STEP 1570 H= 392:30 +CPU= 24.150
  04:00:56 STEP 1571 H= 392:45 +CPU= 22.154
  04:01:26 STEP 1572 H= 393:00 +CPU= 29.637
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001572
  04:01:50 STEP 1573 H= 393:15 +CPU= 23.464
  04:02:12 STEP 1574 H= 393:30 +CPU= 22.560
  04:02:36 STEP 1575 H= 393:45 +CPU= 23.135
  04:03:00 STEP 1576 H= 394:00 +CPU= 24.501
  04:03:25 STEP 1577 H= 394:15 +CPU= 24.377
  04:03:47 STEP 1578 H= 394:30 +CPU= 22.310
  04:04:10 STEP 1579 H= 394:45 +CPU= 22.990
  04:04:34 STEP 1580 H= 395:00 +CPU= 24.226
  04:04:59 STEP 1581 H= 395:15 +CPU= 24.360
  04:05:23 STEP 1582 H= 395:30 +CPU= 24.087
  04:05:46 STEP 1583 H= 395:45 +CPU= 22.855
  04:06:25 STEP 1584 H= 396:00 +CPU= 32.912
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001584
  04:06:49 STEP 1585 H= 396:15 +CPU= 24.140
  04:07:12 STEP 1586 H= 396:30 +CPU= 22.964
  04:07:35 STEP 1587 H= 396:45 +CPU= 22.966
  04:07:58 STEP 1588 H= 397:00 +CPU= 23.010
  04:08:21 STEP 1589 H= 397:15 +CPU= 23.423
  04:08:45 STEP 1590 H= 397:30 +CPU= 23.855
  04:09:08 STEP 1591 H= 397:45 +CPU= 23.283
  04:09:33 STEP 1592 H= 398:00 +CPU= 24.550
  04:09:56 STEP 1593 H= 398:15 +CPU= 22.959
  04:10:23 STEP 1594 H= 398:30 +CPU= 26.534
  04:10:45 STEP 1595 H= 398:45 +CPU= 22.517
  04:11:18 STEP 1596 H= 399:00 +CPU= 29.349
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001596
  04:11:42 STEP 1597 H= 399:15 +CPU= 24.020
  04:12:06 STEP 1598 H= 399:30 +CPU= 23.804
  04:12:29 STEP 1599 H= 399:45 +CPU= 23.106
  04:12:53 STEP 1600 H= 400:00 +CPU= 23.430
  04:13:17 STEP 1601 H= 400:15 +CPU= 23.774
  04:13:39 STEP 1602 H= 400:30 +CPU= 22.712
  04:14:02 STEP 1603 H= 400:45 +CPU= 22.822
  04:14:25 STEP 1604 H= 401:00 +CPU= 22.808
  04:14:49 STEP 1605 H= 401:15 +CPU= 23.911
  04:15:13 STEP 1606 H= 401:30 +CPU= 23.840
  04:15:36 STEP 1607 H= 401:45 +CPU= 22.737
  04:16:05 STEP 1608 H= 402:00 +CPU= 28.887
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001608
  04:16:29 STEP 1609 H= 402:15 +CPU= 24.312
  04:16:53 STEP 1610 H= 402:30 +CPU= 23.639
  04:17:15 STEP 1611 H= 402:45 +CPU= 22.262
  04:17:39 STEP 1612 H= 403:00 +CPU= 23.530
  04:18:02 STEP 1613 H= 403:15 +CPU= 23.797
  04:18:27 STEP 1614 H= 403:30 +CPU= 24.040
  04:18:49 STEP 1615 H= 403:45 +CPU= 22.561
  04:19:12 STEP 1616 H= 404:00 +CPU= 23.154
  04:19:35 STEP 1617 H= 404:15 +CPU= 22.833
  04:20:00 STEP 1618 H= 404:30 +CPU= 24.272
  04:20:25 STEP 1619 H= 404:45 +CPU= 24.822
  04:20:53 STEP 1620 H= 405:00 +CPU= 28.015
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001620
  04:21:16 STEP 1621 H= 405:15 +CPU= 23.084
  04:21:41 STEP 1622 H= 405:30 +CPU= 24.567
  04:22:03 STEP 1623 H= 405:45 +CPU= 22.378
  04:22:26 STEP 1624 H= 406:00 +CPU= 22.847
  04:22:50 STEP 1625 H= 406:15 +CPU= 24.163
  04:23:15 STEP 1626 H= 406:30 +CPU= 23.394
  04:23:37 STEP 1627 H= 406:45 +CPU= 22.269
  04:24:01 STEP 1628 H= 407:00 +CPU= 23.508
  04:24:33 STEP 1629 H= 407:15 +CPU= 25.715
  04:24:57 STEP 1630 H= 407:30 +CPU= 22.348
  04:25:19 STEP 1631 H= 407:45 +CPU= 22.725
  04:25:59 STEP 1632 H= 408:00 +CPU= 32.950
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001632
  04:26:25 STEP 1633 H= 408:15 +CPU= 25.604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001632
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0_r1209853097_16.zip
Uploading the intermediate file: upload_file_16.zip
  04:26:47 STEP 1634 H= 408:30 +CPU= 22.180
  04:27:10 STEP 1635 H= 408:45 +CPU= 22.602
  04:27:34 STEP 1636 H= 409:00 +CPU= 24.266
  04:27:58 STEP 1637 H= 409:15 +CPU= 24.276
  04:28:21 STEP 1638 H= 409:30 +CPU= 22.632
  04:28:44 STEP 1639 H= 409:45 +CPU= 22.710
  04:29:08 STEP 1640 H= 410:00 +CPU= 24.638
  04:29:32 STEP 1641 H= 410:15 +CPU= 23.567
  04:29:55 STEP 1642 H= 410:30 +CPU= 22.512
  04:30:18 STEP 1643 H= 410:45 +CPU= 23.506
  04:30:50 STEP 1644 H= 411:00 +CPU= 31.526
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001644
  04:31:14 STEP 1645 H= 411:15 +CPU= 23.774
  04:31:38 STEP 1646 H= 411:30 +CPU= 24.412
  04:32:02 STEP 1647 H= 411:45 +CPU= 23.581
  04:32:25 STEP 1648 H= 412:00 +CPU= 23.416
  04:32:48 STEP 1649 H= 412:15 +CPU= 22.858
  04:33:13 STEP 1650 H= 412:30 +CPU= 24.749
  04:33:36 STEP 1651 H= 412:45 +CPU= 22.741
  04:33:59 STEP 1652 H= 413:00 +CPU= 22.938
  04:34:22 STEP 1653 H= 413:15 +CPU= 23.209
  04:34:47 STEP 1654 H= 413:30 +CPU= 25.311
  04:35:10 STEP 1655 H= 413:45 +CPU= 23.154
  04:35:39 STEP 1656 H= 414:00 +CPU= 28.206
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001656
  04:36:03 STEP 1657 H= 414:15 +CPU= 23.769
  04:36:27 STEP 1658 H= 414:30 +CPU= 24.131
  04:36:49 STEP 1659 H= 414:45 +CPU= 22.321
  04:37:13 STEP 1660 H= 415:00 +CPU= 23.585
  04:37:36 STEP 1661 H= 415:15 +CPU= 23.510
  04:38:04 STEP 1662 H= 415:30 +CPU= 24.516
  04:38:27 STEP 1663 H= 415:45 +CPU= 22.905
  04:38:50 STEP 1664 H= 416:00 +CPU= 23.710
  04:39:16 STEP 1665 H= 416:15 +CPU= 25.192
  04:39:39 STEP 1666 H= 416:30 +CPU= 23.446
  04:40:02 STEP 1667 H= 416:45 +CPU= 22.796
  04:40:32 STEP 1668 H= 417:00 +CPU= 29.621
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001668
  04:40:57 STEP 1669 H= 417:15 +CPU= 25.128
  04:41:21 STEP 1670 H= 417:30 +CPU= 23.876
  04:41:44 STEP 1671 H= 417:45 +CPU= 22.711
  04:42:07 STEP 1672 H= 418:00 +CPU= 23.441
  04:42:31 STEP 1673 H= 418:15 +CPU= 23.672
  04:42:54 STEP 1674 H= 418:30 +CPU= 22.833
  04:43:18 STEP 1675 H= 418:45 +CPU= 24.488
  04:43:42 STEP 1676 H= 419:00 +CPU= 23.341
  04:44:05 STEP 1677 H= 419:15 +CPU= 23.753
  04:44:29 STEP 1678 H= 419:30 +CPU= 23.186
  04:44:52 STEP 1679 H= 419:45 +CPU= 23.496
  04:45:26 STEP 1680 H= 420:00 +CPU= 33.164
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001680
  04:45:50 STEP 1681 H= 420:15 +CPU= 24.149
  04:46:14 STEP 1682 H= 420:30 +CPU= 24.028
  04:46:37 STEP 1683 H= 420:45 +CPU= 22.916
  04:47:00 STEP 1684 H= 421:00 +CPU= 23.035
  04:47:24 STEP 1685 H= 421:15 +CPU= 23.623
  04:47:47 STEP 1686 H= 421:30 +CPU= 23.428
  04:48:10 STEP 1687 H= 421:45 +CPU= 22.716
  04:48:34 STEP 1688 H= 422:00 +CPU= 24.161
  04:48:58 STEP 1689 H= 422:15 +CPU= 23.664
  04:49:23 STEP 1690 H= 422:30 +CPU= 25.384
  04:49:46 STEP 1691 H= 422:45 +CPU= 22.724
  04:50:16 STEP 1692 H= 423:00 +CPU= 29.840
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001692
  04:50:42 STEP 1693 H= 423:15 +CPU= 26.339
  04:51:06 STEP 1694 H= 423:30 +CPU= 23.415
  04:51:27 STEP 1695 H= 423:45 +CPU= 21.592
  04:51:52 STEP 1696 H= 424:00 +CPU= 23.994
  04:52:17 STEP 1697 H= 424:15 +CPU= 25.878
  04:52:41 STEP 1698 H= 424:30 +CPU= 23.126
  04:53:03 STEP 1699 H= 424:45 +CPU= 22.377
  04:53:27 STEP 1700 H= 425:00 +CPU= 24.369
  04:53:51 STEP 1701 H= 425:15 +CPU= 23.205
  04:54:13 STEP 1702 H= 425:30 +CPU= 22.659
  04:54:36 STEP 1703 H= 425:45 +CPU= 22.961
  04:55:08 STEP 1704 H= 426:00 +CPU= 31.523
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001704
  04:55:32 STEP 1705 H= 426:15 +CPU= 23.992
  04:55:54 STEP 1706 H= 426:30 +CPU= 22.430
  04:56:18 STEP 1707 H= 426:45 +CPU= 23.236
  04:56:43 STEP 1708 H= 427:00 +CPU= 25.086
  04:57:06 STEP 1709 H= 427:15 +CPU= 23.046
  04:57:28 STEP 1710 H= 427:30 +CPU= 22.411
  04:57:52 STEP 1711 H= 427:45 +CPU= 23.229
  04:58:17 STEP 1712 H= 428:00 +CPU= 24.905
  04:58:42 STEP 1713 H= 428:15 +CPU= 22.771
  04:59:05 STEP 1714 H= 428:30 +CPU= 23.020
  04:59:29 STEP 1715 H= 428:45 +CPU= 23.932
  04:59:58 STEP 1716 H= 429:00 +CPU= 28.547
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001716
  05:00:22 STEP 1717 H= 429:15 +CPU= 24.104
  05:00:48 STEP 1718 H= 429:30 +CPU= 25.994
  05:01:12 STEP 1719 H= 429:45 +CPU= 23.838
  05:01:35 STEP 1720 H= 430:00 +CPU= 23.030
  05:01:58 STEP 1721 H= 430:15 +CPU= 23.266
  05:02:23 STEP 1722 H= 430:30 +CPU= 23.971
  05:02:45 STEP 1723 H= 430:45 +CPU= 22.542
  05:03:08 STEP 1724 H= 431:00 +CPU= 23.228
  05:03:34 STEP 1725 H= 431:15 +CPU= 25.883
  05:03:59 STEP 1726 H= 431:30 +CPU= 24.298
  05:04:21 STEP 1727 H= 431:45 +CPU= 22.609
  05:04:54 STEP 1728 H= 432:00 +CPU= 32.792
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001728
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.328992190636342E-06 0.000000000000000E+00 0.155093152346760E-03
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.200144275988973E-06 0.000000000000000E+00 0.287022478683848E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.336140135627090E-07 0.000000000000000E+00 0.557587764961663E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.187425277178745E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.138227077546302E-05 0.000000000000000E+00 0.426915120997543E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.328865824413375E-06 0.000000000000000E+00 0.168429808252476E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.187411939249386E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.154782370036161E-06 0.000000000000000E+00 0.277001560520619E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.221002352103982E-06 0.000000000000000E+00 0.364438468779746E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.871877456163252E-06 0.000000000000000E+00 0.590624151110640E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.655479506563329E-06 0.000000000000000E+00 0.593515510321361E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.787891346387584E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1728 STEPO   F00000000
 CREATE RESTART FILE ./srf00180000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00180000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00180000.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    2.7652 SECONDS******** Mbytes/s, TOTAL TIME=    3.4336(80.5%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00171200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1728 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001728 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001728
 MPI-TASK:   1 -    310965572 BYTES IN      6 RECORDS TRANSFERRED IN    0.0009 SECONDS345517.3 Mbytes/s, TOTAL TIME=    0.0081(11.1%)
 NSTEP =  1728 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001728 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001728
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0149 SECONDS 47476.5 Mbytes/s, TOTAL TIME=    0.2170( 6.9%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001728 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001728
 MPI-TASK:   1 -   4975449152 BYTES IN    125 RECORDS TRANSFERRED IN    0.0094 SECONDS529303.1 Mbytes/s, TOTAL TIME=    0.1766( 5.3%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001728 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001728
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0037( 5.4%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001728 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001728
 MPI-TASK:   1 -    302385843 BYTES IN      4 RECORDS TRANSFERRED IN    0.0005 SECONDS604771.7 Mbytes/s, TOTAL TIME=    0.0058( 8.6%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1728 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1728 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001728 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001728
 MPI-TASK:   1 -    310965572 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS388707.0 Mbytes/s, TOTAL TIME=    0.0101( 7.9%)
 05:04:54 STEP 1728 H= 432:00 +CPU= 32.792
 NSTEP =  1729 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 05:04:22 F00000000 STEPO     1728    22.742   22.742   22.765     77:09     77:51 0.56858968276531E-05       2GB       0MB
 05:04:25 0AA000000 STEPO     1728     3.639    3.639    3.713     77:12     77:54 0.56858968276531E-05       2GB       0MB
 05:04:26 FULLPOS-B DYNFPOS   1728     1.159    1.159    1.159     77:13     77:56 0.56858968276531E-05       2GB       0MB
 05:04:29 FULLPOS-V DYNFPOS   1728     2.726    2.726    2.726     77:16     77:58 0.56858968276531E-05       2GB       0MB
 05:04:30 FULLPOS-T DYNFPOS   1728     0.496    0.496    0.497     77:17     77:59 0.56858968276531E-05       2GB       0MB
 05:04:30 0AAA00AAA STEPO     1728     0.490    0.490    0.490     77:17     77:59 0.56858968276531E-05       2GB       0MB
 05:04:54 0AAA00AAA STEPO     1729    24.151   24.151   24.230     77:41     78:24 0.56858968276531E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1728",
 CTIME   = "00180000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1716, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/14/progress_file_12294048.xml

<running_values>
  <last_cpu_time>42125.980000</last_cpu_time>
  <upload_file_number>17</upload_file_number>
  <last_iter>1729</last_iter>
  <last_upload>1469700</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a4ex
(argv4) batchid: 1018
(argv5) wuid: 12294048
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/14
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/14/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/14/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ce46925d63d805672d59d647ea272b6c to: /var/lib/boinc-client/slots/14/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/14/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048.zip
ic_ancil_file: ic_ancil_12294048
ifsdata_file: ifsdata_12294048
climate_data_file: clim_data_12294048
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/14/ic_ancil_12294048.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/14/ic_ancil_12294048.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bb46729b255b7a125928ef15a43a056c to: /var/lib/boinc-client/slots/14/ifsdata/ifsdata_12294048.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/14/ifsdata/ifsdata_12294048.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/14/159l_2/clim_data_12294048.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/14/159l_2/clim_data_12294048.zip
Checking for progress XML file: /var/lib/boinc-client/slots/14/progress_file_12294048.xml
Opened progress file ok : /var/lib/boinc-client/slots/14/progress_file_12294048.xml
-- Model is restarting --
Adjusting last_iter, 1729, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/14/progress_file_12294048.xml
last_cpu_time: 42125
upload_file_number: 17
last_iter: 1729
last_upload: 1469700
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0_r1209853097
The child process has been launched with process id: 26046
Executing the command: /var/lib/boinc-client/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:26046:26046] [20240618:050617:1718679977:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001728
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0_r1209853097_17.zip
  05:06:57 STEP 1728 H= 432:00 +CPU= 26.279
Uploading the intermediate file: upload_file_17.zip
  05:07:19 STEP 1729 H= 432:15 +CPU= 22.132
  05:07:41 STEP 1730 H= 432:30 +CPU= 22.328
  05:08:08 STEP 1731 H= 432:45 +CPU= 24.629
  05:08:33 STEP 1732 H= 433:00 +CPU= 23.961
  05:08:56 STEP 1733 H= 433:15 +CPU= 22.534
  05:09:19 STEP 1734 H= 433:30 +CPU= 23.092
  05:09:45 STEP 1735 H= 433:45 +CPU= 26.143
  05:10:11 STEP 1736 H= 434:00 +CPU= 24.397
  05:10:34 STEP 1737 H= 434:15 +CPU= 22.681
  05:10:57 STEP 1738 H= 434:30 +CPU= 22.761
  05:11:23 STEP 1739 H= 434:45 +CPU= 26.496
  05:11:52 STEP 1740 H= 435:00 +CPU= 28.367
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001740
  05:12:15 STEP 1741 H= 435:15 +CPU= 22.336
  05:12:39 STEP 1742 H= 435:30 +CPU= 24.550
  05:13:02 STEP 1743 H= 435:45 +CPU= 22.990
  05:13:26 STEP 1744 H= 436:00 +CPU= 23.464
  05:13:49 STEP 1745 H= 436:15 +CPU= 23.176
  05:14:14 STEP 1746 H= 436:30 +CPU= 25.286
  05:14:37 STEP 1747 H= 436:45 +CPU= 23.180
  05:15:01 STEP 1748 H= 437:00 +CPU= 23.145
  05:15:26 STEP 1749 H= 437:15 +CPU= 24.994
  05:15:50 STEP 1750 H= 437:30 +CPU= 24.598
  05:16:15 STEP 1751 H= 437:45 +CPU= 24.787
  05:16:44 STEP 1752 H= 438:00 +CPU= 28.419
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001752
  05:17:07 STEP 1753 H= 438:15 +CPU= 23.370
  05:17:31 STEP 1754 H= 438:30 +CPU= 23.685
  05:17:55 STEP 1755 H= 438:45 +CPU= 23.717
  05:18:18 STEP 1756 H= 439:00 +CPU= 23.190
  05:18:40 STEP 1757 H= 439:15 +CPU= 22.471
  05:19:04 STEP 1758 H= 439:30 +CPU= 23.283
  05:19:29 STEP 1759 H= 439:45 +CPU= 25.208
  05:19:52 STEP 1760 H= 440:00 +CPU= 22.890
  05:20:15 STEP 1761 H= 440:15 +CPU= 22.418
  05:20:40 STEP 1762 H= 440:30 +CPU= 24.972
  05:21:05 STEP 1763 H= 440:45 +CPU= 24.928
  05:21:34 STEP 1764 H= 441:00 +CPU= 28.801
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001764
  05:21:57 STEP 1765 H= 441:15 +CPU= 23.181
  05:22:22 STEP 1766 H= 441:30 +CPU= 25.487
  05:22:45 STEP 1767 H= 441:45 +CPU= 22.672
  05:23:08 STEP 1768 H= 442:00 +CPU= 23.288
  05:23:33 STEP 1769 H= 442:15 +CPU= 24.389
  05:23:57 STEP 1770 H= 442:30 +CPU= 24.223
  05:24:20 STEP 1771 H= 442:45 +CPU= 22.527
  05:24:43 STEP 1772 H= 443:00 +CPU= 23.020
  05:25:07 STEP 1773 H= 443:15 +CPU= 24.307
  05:25:33 STEP 1774 H= 443:30 +CPU= 25.596
  05:25:56 STEP 1775 H= 443:45 +CPU= 22.800
  05:26:28 STEP 1776 H= 444:00 +CPU= 32.313
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001776
  05:26:53 STEP 1777 H= 444:15 +CPU= 24.566
  05:27:16 STEP 1778 H= 444:30 +CPU= 23.087
  05:27:39 STEP 1779 H= 444:45 +CPU= 22.797
  05:28:04 STEP 1780 H= 445:00 +CPU= 24.633
  05:28:29 STEP 1781 H= 445:15 +CPU= 24.972
  05:28:58 STEP 1782 H= 445:30 +CPU= 22.588
  05:29:22 STEP 1783 H= 445:45 +CPU= 23.205
  05:29:47 STEP 1784 H= 446:00 +CPU= 25.419
  05:30:11 STEP 1785 H= 446:15 +CPU= 24.006
  05:30:35 STEP 1786 H= 446:30 +CPU= 23.892
  05:30:59 STEP 1787 H= 446:45 +CPU= 23.773
  05:31:29 STEP 1788 H= 447:00 +CPU= 30.065
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001788
  05:31:51 STEP 1789 H= 447:15 +CPU= 22.331
  05:32:15 STEP 1790 H= 447:30 +CPU= 23.629
  05:32:38 STEP 1791 H= 447:45 +CPU= 23.423
  05:33:02 STEP 1792 H= 448:00 +CPU= 23.286
  05:33:25 STEP 1793 H= 448:15 +CPU= 22.764
  05:33:48 STEP 1794 H= 448:30 +CPU= 23.385
  05:34:12 STEP 1795 H= 448:45 +CPU= 23.589
  05:34:34 STEP 1796 H= 449:00 +CPU= 22.755
  05:34:57 STEP 1797 H= 449:15 +CPU= 22.613
  05:35:25 STEP 1798 H= 449:30 +CPU= 27.338
  05:35:48 STEP 1799 H= 449:45 +CPU= 23.475
  05:36:17 STEP 1800 H= 450:00 +CPU= 28.482
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001800
  05:36:40 STEP 1801 H= 450:15 +CPU= 22.975
  05:37:07 STEP 1802 H= 450:30 +CPU= 25.082
  05:37:32 STEP 1803 H= 450:45 +CPU= 24.156
  05:37:55 STEP 1804 H= 451:00 +CPU= 22.941
  05:38:18 STEP 1805 H= 451:15 +CPU= 23.491
  05:38:43 STEP 1806 H= 451:30 +CPU= 24.568
  05:39:07 STEP 1807 H= 451:45 +CPU= 24.518
  05:39:32 STEP 1808 H= 452:00 +CPU= 24.229
  05:39:56 STEP 1809 H= 452:15 +CPU= 24.714
  05:40:21 STEP 1810 H= 452:30 +CPU= 24.202
  05:40:42 STEP 1811 H= 452:45 +CPU= 21.735
  05:41:13 STEP 1812 H= 453:00 +CPU= 30.181
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001812
  05:41:38 STEP 1813 H= 453:15 +CPU= 25.417
  05:42:01 STEP 1814 H= 453:30 +CPU= 23.155
  05:42:24 STEP 1815 H= 453:45 +CPU= 22.628
  05:42:49 STEP 1816 H= 454:00 +CPU= 24.765
  05:43:14 STEP 1817 H= 454:15 +CPU= 24.985
  05:43:37 STEP 1818 H= 454:30 +CPU= 23.141
  05:44:00 STEP 1819 H= 454:45 +CPU= 22.523
  05:44:23 STEP 1820 H= 455:00 +CPU= 23.832
  05:44:48 STEP 1821 H= 455:15 +CPU= 24.581
  05:45:12 STEP 1822 H= 455:30 +CPU= 23.365
  05:45:34 STEP 1823 H= 455:45 +CPU= 22.777
  05:46:09 STEP 1824 H= 456:00 +CPU= 33.111
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001824
  05:46:33 STEP 1825 H= 456:15 +CPU= 24.268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001824
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0_r1209853097_18.zip
Uploading the intermediate file: upload_file_18.zip
  05:46:57 STEP 1826 H= 456:30 +CPU= 23.956
  05:47:21 STEP 1827 H= 456:45 +CPU= 23.741
  05:47:45 STEP 1828 H= 457:00 +CPU= 23.542
  05:48:08 STEP 1829 H= 457:15 +CPU= 23.749
  05:48:32 STEP 1830 H= 457:30 +CPU= 23.410
  05:48:55 STEP 1831 H= 457:45 +CPU= 22.776
  05:49:19 STEP 1832 H= 458:00 +CPU= 24.660
  05:49:44 STEP 1833 H= 458:15 +CPU= 25.010
  05:50:08 STEP 1834 H= 458:30 +CPU= 23.186
  05:50:31 STEP 1835 H= 458:45 +CPU= 23.116
  05:51:00 STEP 1836 H= 459:00 +CPU= 29.288
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001836
  05:51:23 STEP 1837 H= 459:15 +CPU= 22.771
  05:51:46 STEP 1838 H= 459:30 +CPU= 22.901
  05:52:10 STEP 1839 H= 459:45 +CPU= 23.916
  05:52:35 STEP 1840 H= 460:00 +CPU= 24.799
  05:52:57 STEP 1841 H= 460:15 +CPU= 22.678
  05:53:21 STEP 1842 H= 460:30 +CPU= 22.317
  05:53:44 STEP 1843 H= 460:45 +CPU= 23.605
  05:54:08 STEP 1844 H= 461:00 +CPU= 24.044
  05:54:31 STEP 1845 H= 461:15 +CPU= 22.760
  05:54:53 STEP 1846 H= 461:30 +CPU= 22.491
  05:55:18 STEP 1847 H= 461:45 +CPU= 24.180
  05:55:50 STEP 1848 H= 462:00 +CPU= 30.168
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001848
  05:56:13 STEP 1849 H= 462:15 +CPU= 22.444
  05:56:35 STEP 1850 H= 462:30 +CPU= 22.510
  05:56:58 STEP 1851 H= 462:45 +CPU= 23.203
  05:57:22 STEP 1852 H= 463:00 +CPU= 23.716
  05:57:45 STEP 1853 H= 463:15 +CPU= 22.812
  05:58:08 STEP 1854 H= 463:30 +CPU= 22.687
  05:58:31 STEP 1855 H= 463:45 +CPU= 23.021
  05:58:55 STEP 1856 H= 464:00 +CPU= 23.868
  05:59:17 STEP 1857 H= 464:15 +CPU= 22.331
  05:59:40 STEP 1858 H= 464:30 +CPU= 22.576
  06:00:03 STEP 1859 H= 464:45 +CPU= 23.549
  06:00:33 STEP 1860 H= 465:00 +CPU= 29.788
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001860
  06:00:56 STEP 1861 H= 465:15 +CPU= 22.818
  06:01:19 STEP 1862 H= 465:30 +CPU= 23.223
  06:01:43 STEP 1863 H= 465:45 +CPU= 23.780
  06:02:06 STEP 1864 H= 466:00 +CPU= 23.346
  06:02:29 STEP 1865 H= 466:15 +CPU= 23.125
  06:02:53 STEP 1866 H= 466:30 +CPU= 23.396
  06:03:17 STEP 1867 H= 466:45 +CPU= 24.404
  06:03:41 STEP 1868 H= 467:00 +CPU= 23.345
  06:04:03 STEP 1869 H= 467:15 +CPU= 22.344
  06:04:27 STEP 1870 H= 467:30 +CPU= 23.705
  06:04:50 STEP 1871 H= 467:45 +CPU= 23.311
  06:05:28 STEP 1872 H= 468:00 +CPU= 34.567
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001872
  06:05:51 STEP 1873 H= 468:15 +CPU= 22.863
  06:06:13 STEP 1874 H= 468:30 +CPU= 22.539
  06:06:35 STEP 1875 H= 468:45 +CPU= 21.515
  06:06:58 STEP 1876 H= 469:00 +CPU= 23.525
  06:07:21 STEP 1877 H= 469:15 +CPU= 22.830
  06:07:45 STEP 1878 H= 469:30 +CPU= 23.453
  06:08:08 STEP 1879 H= 469:45 +CPU= 22.834
  06:08:31 STEP 1880 H= 470:00 +CPU= 23.591
  06:08:55 STEP 1881 H= 470:15 +CPU= 23.729
  06:09:19 STEP 1882 H= 470:30 +CPU= 23.510
  06:09:41 STEP 1883 H= 470:45 +CPU= 22.237
  06:10:10 STEP 1884 H= 471:00 +CPU= 28.511
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001884
  06:10:34 STEP 1885 H= 471:15 +CPU= 24.711
  06:10:57 STEP 1886 H= 471:30 +CPU= 22.105
  06:11:20 STEP 1887 H= 471:45 +CPU= 23.341
  06:11:44 STEP 1888 H= 472:00 +CPU= 24.013
  06:12:08 STEP 1889 H= 472:15 +CPU= 23.587
  06:12:30 STEP 1890 H= 472:30 +CPU= 22.660
  06:12:53 STEP 1891 H= 472:45 +CPU= 22.944
  06:13:17 STEP 1892 H= 473:00 +CPU= 23.698
  06:13:40 STEP 1893 H= 473:15 +CPU= 22.874
  06:14:02 STEP 1894 H= 473:30 +CPU= 22.208
  06:14:26 STEP 1895 H= 473:45 +CPU= 23.396
  06:14:54 STEP 1896 H= 474:00 +CPU= 28.604
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001896
  06:15:17 STEP 1897 H= 474:15 +CPU= 23.136
  06:15:41 STEP 1898 H= 474:30 +CPU= 23.111
  06:16:03 STEP 1899 H= 474:45 +CPU= 22.542
  06:16:26 STEP 1900 H= 475:00 +CPU= 22.795
  06:16:50 STEP 1901 H= 475:15 +CPU= 23.577
  06:17:12 STEP 1902 H= 475:30 +CPU= 22.474
  06:17:35 STEP 1903 H= 475:45 +CPU= 22.536
  06:17:58 STEP 1904 H= 476:00 +CPU= 23.103
  06:18:21 STEP 1905 H= 476:15 +CPU= 22.978
  06:18:43 STEP 1906 H= 476:30 +CPU= 22.371
  06:19:06 STEP 1907 H= 476:45 +CPU= 22.801
  06:19:37 STEP 1908 H= 477:00 +CPU= 29.789
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001908
  06:19:59 STEP 1909 H= 477:15 +CPU= 22.272
  06:20:22 STEP 1910 H= 477:30 +CPU= 23.164
  06:20:46 STEP 1911 H= 477:45 +CPU= 23.277
  06:21:10 STEP 1912 H= 478:00 +CPU= 24.271
  06:21:32 STEP 1913 H= 478:15 +CPU= 22.439
  06:21:55 STEP 1914 H= 478:30 +CPU= 22.508
  06:22:17 STEP 1915 H= 478:45 +CPU= 22.214
  06:22:40 STEP 1916 H= 479:00 +CPU= 22.701
  06:23:02 STEP 1917 H= 479:15 +CPU= 22.403
  06:23:26 STEP 1918 H= 479:30 +CPU= 23.530
  06:23:48 STEP 1919 H= 479:45 +CPU= 22.544
  06:24:21 STEP 1920 H= 480:00 +CPU= 30.192
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1501370096E-20      I       0.1316213963E-05      I
ICloud fractionI      0.0000000000E+00      I       0.6818216043E-01      I
I  Relat. Hum. I      0.1399066246E+00      I       0.3929252479E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1058303139E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    295773508 BYTES IN      7 RECORDS TRANSFERRED IN    0.0018 SECONDS164318.6 Mbytes/s, TOTAL TIME=    0.0197( 9.1%)
 06:24:21 STEP 1920 H= 480:00 +CPU= 30.192
 END CNT3
 NSTEP =  1920 CNT0   000000000  21.609  21.609  26.454   0.817               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         5720/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    103    0       46      1044068     5720 ************    25.5    11.3    14.2    12.1    24.9
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 4686970.3 4686970.3    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 4670699.8 4670699.8     99.65      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209    4083.4    4083.4     18.21      0.00
   9 SPCM     - SPECTRAL COMP.                     192     653.3     653.3      2.68      0.00
  10 SCAN2M - PHYSICS                              193   11248.0   11248.0     46.32      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16    1129.1    1129.1      0.39      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  9      47.9      47.9      0.01      0.00
  17 GRIDFPOS IN CNT4                               16      30.5      30.5      0.01      0.00
  27 RESTART FILES - WRITING                         4    4500.1    4500.1      0.38      0.00
  28 RESTART FILES - READING                         1   12997.5   12997.5      0.28      0.00
  29 SU4FPOS IN CNT4                                16       1.4       1.4      0.00      0.00
  30 DYNFPOS IN CNT4                                16    5107.5    5107.5      1.74      0.00
  31 POSDDH IN STEPO                                 1    2246.8    2246.8      0.05      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193    5149.8    5149.8     21.21      0.00
  39 SU0YOMB                                         1     432.8     432.8      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      20.4      20.4      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192     309.6     309.6      1.27      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192      49.0      49.0      0.20      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  9       2.8       2.8      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      420     313.0     313.0      2.80      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       400     174.5     174.5      1.49      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        400      65.8      65.8      0.56      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       420      85.0      85.0      0.76      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      33.8      33.8      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            420      62.4      62.4      0.56      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            400      32.1      32.1      0.27      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            420      58.0      58.0      0.52      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            400      33.1      33.1      0.28      0.00
 400 GSTATS                                      23311       0.0       0.0      0.01      0.00
 401 GSTATS HOOK                                 21635       0.1       0.1      0.05      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     4687.0 CPU TIME    4641.4 VECTOR TIME     4641.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      370.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/14/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/14/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/14/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294048/ICMGGh7zg+000384
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0_r1209853097_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
06:28:01 (26043): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
18 Jun 2024 04:49:57 1469580 22447665 oifs_43r3_bl_a4ex_2016092300_20_1018_12294048_0 1,727,100 46,766 0.0271


©2024 climateprediction.net