climateprediction.net home page
Task 22446902

Task 22446902

Name oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0
Workunit 12293290
Created 13 Jun 2024, 3:12:22 UTC
Sent 14 Jun 2024, 6:20:51 UTC
Report deadline 13 Aug 2024, 6:20:51 UTC
Received 16 Jun 2024, 10:34:48 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1496046
Run time 14 hours 43 min 34 sec
CPU time 14 hours 43 min 34 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.56 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.40 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,482.21 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
s/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001476
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001476
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001476
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001488
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001488
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001488
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001500
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001500
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001500
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001512
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001512
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001512
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001524
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001524
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001524
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307_15.zip
Uploading the intermediate file: upload_file_15.zip
  02:11:22 STEP 1537 H= 384:15 +CPU= 25.210
  02:11:47 STEP 1538 H= 384:30 +CPU= 24.615
  02:12:13 STEP 1539 H= 384:45 +CPU= 25.896
  02:12:37 STEP 1540 H= 385:00 +CPU= 23.951
  02:13:02 STEP 1541 H= 385:15 +CPU= 25.269
  02:13:28 STEP 1542 H= 385:30 +CPU= 25.655
  02:13:53 STEP 1543 H= 385:45 +CPU= 24.620
  02:14:17 STEP 1544 H= 386:00 +CPU= 24.711
  02:14:41 STEP 1545 H= 386:15 +CPU= 23.053
  02:15:04 STEP 1546 H= 386:30 +CPU= 23.034
  02:15:29 STEP 1547 H= 386:45 +CPU= 25.023
  02:15:59 STEP 1548 H= 387:00 +CPU= 30.731
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001548
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001548
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001548
  02:16:24 STEP 1549 H= 387:15 +CPU= 24.664
  02:16:49 STEP 1550 H= 387:30 +CPU= 24.322
  02:17:14 STEP 1551 H= 387:45 +CPU= 24.935
  02:17:40 STEP 1552 H= 388:00 +CPU= 26.217
  02:18:06 STEP 1553 H= 388:15 +CPU= 26.122
  02:18:32 STEP 1554 H= 388:30 +CPU= 26.041
  02:18:57 STEP 1555 H= 388:45 +CPU= 25.186
  02:19:23 STEP 1556 H= 389:00 +CPU= 25.965
  02:19:49 STEP 1557 H= 389:15 +CPU= 25.231
  02:20:13 STEP 1558 H= 389:30 +CPU= 24.525
  02:20:38 STEP 1559 H= 389:45 +CPU= 24.459
  02:21:10 STEP 1560 H= 390:00 +CPU= 32.260
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001560
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001560
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001560
  02:21:35 STEP 1561 H= 390:15 +CPU= 24.969
  02:22:01 STEP 1562 H= 390:30 +CPU= 25.513
  02:22:25 STEP 1563 H= 390:45 +CPU= 24.218
  02:22:48 STEP 1564 H= 391:00 +CPU= 23.494
  02:23:14 STEP 1565 H= 391:15 +CPU= 25.237
  02:23:38 STEP 1566 H= 391:30 +CPU= 23.794
  02:24:02 STEP 1567 H= 391:45 +CPU= 24.889
  02:24:29 STEP 1568 H= 392:00 +CPU= 26.365
  02:24:53 STEP 1569 H= 392:15 +CPU= 23.758
  02:25:18 STEP 1570 H= 392:30 +CPU= 24.935
  02:25:42 STEP 1571 H= 392:45 +CPU= 24.387
  02:26:15 STEP 1572 H= 393:00 +CPU= 32.456
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001572
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001572
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001572
  02:26:39 STEP 1573 H= 393:15 +CPU= 24.616
  02:27:05 STEP 1574 H= 393:30 +CPU= 25.414
  02:27:30 STEP 1575 H= 393:45 +CPU= 25.534
  02:27:56 STEP 1576 H= 394:00 +CPU= 25.414
  02:28:22 STEP 1577 H= 394:15 +CPU= 25.756
  02:28:47 STEP 1578 H= 394:30 +CPU= 25.462
  02:29:12 STEP 1579 H= 394:45 +CPU= 24.581
  02:29:37 STEP 1580 H= 395:00 +CPU= 25.013
  02:30:02 STEP 1581 H= 395:15 +CPU= 24.415
  02:30:27 STEP 1582 H= 395:30 +CPU= 24.984
  02:30:53 STEP 1583 H= 395:45 +CPU= 26.000
  02:31:29 STEP 1584 H= 396:00 +CPU= 36.008
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001584
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001584
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001584
  02:31:54 STEP 1585 H= 396:15 +CPU= 25.246
  02:32:18 STEP 1586 H= 396:30 +CPU= 23.596
  02:32:42 STEP 1587 H= 396:45 +CPU= 23.988
  02:33:08 STEP 1588 H= 397:00 +CPU= 26.471
  02:33:34 STEP 1589 H= 397:15 +CPU= 25.661
  02:34:00 STEP 1590 H= 397:30 +CPU= 25.812
  02:34:24 STEP 1591 H= 397:45 +CPU= 24.571
  02:34:50 STEP 1592 H= 398:00 +CPU= 25.535
  02:35:16 STEP 1593 H= 398:15 +CPU= 25.454
  02:35:40 STEP 1594 H= 398:30 +CPU= 24.663
  02:36:05 STEP 1595 H= 398:45 +CPU= 24.617
  02:36:37 STEP 1596 H= 399:00 +CPU= 32.184
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001596
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001596
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001596
  02:37:02 STEP 1597 H= 399:15 +CPU= 24.967
  02:37:27 STEP 1598 H= 399:30 +CPU= 24.431
  02:37:52 STEP 1599 H= 399:45 +CPU= 25.027
  02:38:18 STEP 1600 H= 400:00 +CPU= 26.472
  02:38:42 STEP 1601 H= 400:15 +CPU= 23.824
  02:39:07 STEP 1602 H= 400:30 +CPU= 24.948
  02:39:33 STEP 1603 H= 400:45 +CPU= 25.447
  02:39:56 STEP 1604 H= 401:00 +CPU= 23.698
  02:40:21 STEP 1605 H= 401:15 +CPU= 24.336
  02:40:45 STEP 1606 H= 401:30 +CPU= 24.443
  02:41:10 STEP 1607 H= 401:45 +CPU= 24.752
  02:41:40 STEP 1608 H= 402:00 +CPU= 29.983
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001608
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001608
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001608
  02:42:05 STEP 1609 H= 402:15 +CPU= 24.329
  02:42:29 STEP 1610 H= 402:30 +CPU= 24.687
  02:42:54 STEP 1611 H= 402:45 +CPU= 24.733
  02:43:20 STEP 1612 H= 403:00 +CPU= 26.317
  02:43:46 STEP 1613 H= 403:15 +CPU= 25.538
  02:44:11 STEP 1614 H= 403:30 +CPU= 24.873
  02:44:35 STEP 1615 H= 403:45 +CPU= 24.073
  02:45:02 STEP 1616 H= 404:00 +CPU= 26.991
  02:45:28 STEP 1617 H= 404:15 +CPU= 26.084
  02:45:54 STEP 1618 H= 404:30 +CPU= 25.754
  02:46:20 STEP 1619 H= 404:45 +CPU= 25.529
  02:46:53 STEP 1620 H= 405:00 +CPU= 32.828
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001620
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001620
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001620
  02:47:18 STEP 1621 H= 405:15 +CPU= 25.465
  02:47:43 STEP 1622 H= 405:30 +CPU= 25.199
  02:48:08 STEP 1623 H= 405:45 +CPU= 25.123
  02:48:35 STEP 1624 H= 406:00 +CPU= 26.241
  02:49:01 STEP 1625 H= 406:15 +CPU= 25.694
  02:49:26 STEP 1626 H= 406:30 +CPU= 25.868
  02:49:52 STEP 1627 H= 406:45 +CPU= 25.685
  02:50:19 STEP 1628 H= 407:00 +CPU= 26.533
  02:50:43 STEP 1629 H= 407:15 +CPU= 24.102
  02:51:08 STEP 1630 H= 407:30 +CPU= 25.199
  02:51:34 STEP 1631 H= 407:45 +CPU= 25.902
  02:52:11 STEP 1632 H= 408:00 +CPU= 36.327
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001632
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001632
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001536
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001548
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001560
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001572
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001584
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001596
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001608
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001620
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001620
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001620
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307_16.zip
Uploading the intermediate file: upload_file_16.zip
  02:52:37 STEP 1633 H= 408:15 +CPU= 26.300
  02:53:02 STEP 1634 H= 408:30 +CPU= 25.199
  02:53:27 STEP 1635 H= 408:45 +CPU= 24.793
  02:53:54 STEP 1636 H= 409:00 +CPU= 26.567
  02:54:20 STEP 1637 H= 409:15 +CPU= 25.841
  02:54:46 STEP 1638 H= 409:30 +CPU= 26.200
  02:55:12 STEP 1639 H= 409:45 +CPU= 25.534
  02:55:38 STEP 1640 H= 410:00 +CPU= 26.398
  02:56:04 STEP 1641 H= 410:15 +CPU= 25.575
  02:56:30 STEP 1642 H= 410:30 +CPU= 26.410
  02:56:56 STEP 1643 H= 410:45 +CPU= 25.968
  02:57:27 STEP 1644 H= 411:00 +CPU= 31.352
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001644
  02:57:54 STEP 1645 H= 411:15 +CPU= 26.015
  02:58:19 STEP 1646 H= 411:30 +CPU= 25.831
  02:58:45 STEP 1647 H= 411:45 +CPU= 25.349
  02:59:11 STEP 1648 H= 412:00 +CPU= 25.864
  02:59:36 STEP 1649 H= 412:15 +CPU= 25.374
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3tv
(argv4) batchid: 1018
(argv5) wuid: 12293290
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /boinc_data/boinc/slots/7
Project directory is: /boinc_data/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290
..mkdir for temp folder for results failed
Copying: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /boinc_data/boinc/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /boinc_data/boinc/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ee46bccdd5523a26a1e57cd692e52d0d to: /boinc_data/boinc/slots/7/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290.zip
Unzipping the namelist zip file: /boinc_data/boinc/slots/7/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290.zip
ic_ancil_file: ic_ancil_12293290
ifsdata_file: ifsdata_12293290
climate_data_file: clim_data_12293290
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /boinc_data/boinc/slots/7/ic_ancil_12293290.zip
Unzipping the IC ancils zip file: /boinc_data/boinc/slots/7/ic_ancil_12293290.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_09119fb5b6f9a2521551e0fe321917de to: /boinc_data/boinc/slots/7/ifsdata/ifsdata_12293290.zip
Unzipping the ifsdata_zip file: /boinc_data/boinc/slots/7/ifsdata/ifsdata_12293290.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /boinc_data/boinc/slots/7/159l_2/clim_data_12293290.zip
Unzipping the climate data zip file: /boinc_data/boinc/slots/7/159l_2/clim_data_12293290.zip
Checking for progress XML file: /boinc_data/boinc/slots/7/progress_file_12293290.xml
Opened progress file ok : /boinc_data/boinc/slots/7/progress_file_12293290.xml
-- Model is restarting --
Adjusting last_iter, 1650, to previous model restart step.
Creating progress file: /boinc_data/boinc/slots/7/progress_file_12293290.xml
last_cpu_time: 45868
upload_file_number: 17
last_iter: 1633
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307
The child process has been launched with process id: 871886
Executing the command: /boinc_data/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:871886:871886] [20240616:030809:1718525289:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:08:37 STEP 1632 H= 408:00 +CPU= 24.507
  03:08:59 STEP 1633 H= 408:15 +CPU= 21.948
  03:09:22 STEP 1634 H= 408:30 +CPU= 22.657
  03:09:45 STEP 1635 H= 408:45 +CPU= 22.653
  03:10:08 STEP 1636 H= 409:00 +CPU= 23.218
  03:10:25 STEP 1637 H= 409:15 +CPU= 17.297
  03:10:43 STEP 1638 H= 409:30 +CPU= 17.947
  03:11:07 STEP 1639 H= 409:45 +CPU= 23.143
  03:11:32 STEP 1640 H= 410:00 +CPU= 25.557
  03:11:57 STEP 1641 H= 410:15 +CPU= 24.809
  03:12:23 STEP 1642 H= 410:30 +CPU= 25.532
  03:12:48 STEP 1643 H= 410:45 +CPU= 25.059
  03:13:19 STEP 1644 H= 411:00 +CPU= 31.277
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001644
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001644
  03:13:44 STEP 1645 H= 411:15 +CPU= 24.704
  03:14:09 STEP 1646 H= 411:30 +CPU= 25.539
  03:14:34 STEP 1647 H= 411:45 +CPU= 24.900
  03:15:01 STEP 1648 H= 412:00 +CPU= 26.409
  03:15:26 STEP 1649 H= 412:15 +CPU= 25.491
  03:15:51 STEP 1650 H= 412:30 +CPU= 24.537
  03:16:16 STEP 1651 H= 412:45 +CPU= 25.572
  03:16:42 STEP 1652 H= 413:00 +CPU= 25.530
  03:17:07 STEP 1653 H= 413:15 +CPU= 25.375
  03:17:32 STEP 1654 H= 413:30 +CPU= 24.872
  03:17:57 STEP 1655 H= 413:45 +CPU= 24.565
  03:18:29 STEP 1656 H= 414:00 +CPU= 32.325
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001656
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001656
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001656
  03:18:54 STEP 1657 H= 414:15 +CPU= 24.878
  03:19:19 STEP 1658 H= 414:30 +CPU= 25.175
  03:19:45 STEP 1659 H= 414:45 +CPU= 25.148
  03:20:10 STEP 1660 H= 415:00 +CPU= 24.930
  03:20:35 STEP 1661 H= 415:15 +CPU= 25.147
  03:20:59 STEP 1662 H= 415:30 +CPU= 23.896
  03:21:22 STEP 1663 H= 415:45 +CPU= 23.722
  03:21:48 STEP 1664 H= 416:00 +CPU= 25.201
  03:22:13 STEP 1665 H= 416:15 +CPU= 24.925
  03:22:37 STEP 1666 H= 416:30 +CPU= 24.187
  03:23:01 STEP 1667 H= 416:45 +CPU= 24.482
  03:23:32 STEP 1668 H= 417:00 +CPU= 30.697
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001668
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001668
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001668
  03:23:52 STEP 1669 H= 417:15 +CPU= 19.683
  03:24:10 STEP 1670 H= 417:30 +CPU= 17.947
  03:24:30 STEP 1671 H= 417:45 +CPU= 19.686
  03:24:56 STEP 1672 H= 418:00 +CPU= 25.851
  03:25:20 STEP 1673 H= 418:15 +CPU= 24.724
  03:25:45 STEP 1674 H= 418:30 +CPU= 24.314
  03:26:10 STEP 1675 H= 418:45 +CPU= 25.202
  03:26:36 STEP 1676 H= 419:00 +CPU= 26.435
  03:27:00 STEP 1677 H= 419:15 +CPU= 23.278
  03:27:25 STEP 1678 H= 419:30 +CPU= 25.028
  03:27:49 STEP 1679 H= 419:45 +CPU= 24.382
  03:28:23 STEP 1680 H= 420:00 +CPU= 33.435
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001680
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001680
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001680
  03:28:48 STEP 1681 H= 420:15 +CPU= 25.453
  03:29:12 STEP 1682 H= 420:30 +CPU= 23.633
  03:29:37 STEP 1683 H= 420:45 +CPU= 25.270
  03:30:02 STEP 1684 H= 421:00 +CPU= 25.284
  03:30:28 STEP 1685 H= 421:15 +CPU= 25.581
  03:30:52 STEP 1686 H= 421:30 +CPU= 24.250
  03:31:17 STEP 1687 H= 421:45 +CPU= 24.080
  03:31:41 STEP 1688 H= 422:00 +CPU= 24.280
  03:32:06 STEP 1689 H= 422:15 +CPU= 25.286
  03:32:31 STEP 1690 H= 422:30 +CPU= 25.104
  03:32:57 STEP 1691 H= 422:45 +CPU= 25.430
  03:33:28 STEP 1692 H= 423:00 +CPU= 30.904
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001692
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001692
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001692
  03:33:54 STEP 1693 H= 423:15 +CPU= 25.811
  03:34:19 STEP 1694 H= 423:30 +CPU= 25.485
  03:34:43 STEP 1695 H= 423:45 +CPU= 24.208
  03:35:09 STEP 1696 H= 424:00 +CPU= 25.873
  03:35:33 STEP 1697 H= 424:15 +CPU= 23.701
  03:35:58 STEP 1698 H= 424:30 +CPU= 24.624
  03:36:23 STEP 1699 H= 424:45 +CPU= 25.250
  03:36:48 STEP 1700 H= 425:00 +CPU= 25.509
  03:37:14 STEP 1701 H= 425:15 +CPU= 25.226
  03:37:38 STEP 1702 H= 425:30 +CPU= 23.765
  03:38:02 STEP 1703 H= 425:45 +CPU= 23.917
  03:38:33 STEP 1704 H= 426:00 +CPU= 31.383
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001704
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001704
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001704
  03:38:59 STEP 1705 H= 426:15 +CPU= 25.486
  03:39:23 STEP 1706 H= 426:30 +CPU= 24.209
  03:39:47 STEP 1707 H= 426:45 +CPU= 24.454
  03:40:13 STEP 1708 H= 427:00 +CPU= 25.760
  03:40:38 STEP 1709 H= 427:15 +CPU= 24.497
  03:41:03 STEP 1710 H= 427:30 +CPU= 24.902
  03:41:28 STEP 1711 H= 427:45 +CPU= 25.291
  03:41:53 STEP 1712 H= 428:00 +CPU= 25.449
  03:42:18 STEP 1713 H= 428:15 +CPU= 24.273
  03:42:36 STEP 1714 H= 428:30 +CPU= 17.916
  03:42:53 STEP 1715 H= 428:45 +CPU= 17.349
  03:43:20 STEP 1716 H= 429:00 +CPU= 27.260
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001716
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001716
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001716
  03:43:45 STEP 1717 H= 429:15 +CPU= 24.264
  03:44:09 STEP 1718 H= 429:30 +CPU= 24.432
  03:44:34 STEP 1719 H= 429:45 +CPU= 24.868
  03:45:00 STEP 1720 H= 430:00 +CPU= 25.306
  03:45:24 STEP 1721 H= 430:15 +CPU= 24.366
  03:45:50 STEP 1722 H= 430:30 +CPU= 25.661
  03:46:14 STEP 1723 H= 430:45 +CPU= 24.525
  03:46:40 STEP 1724 H= 431:00 +CPU= 25.353
  03:47:05 STEP 1725 H= 431:15 +CPU= 25.121
  03:47:30 STEP 1726 H= 431:30 +CPU= 25.337
  03:47:55 STEP 1727 H= 431:45 +CPU= 24.819
  03:48:30 STEP 1728 H= 432:00 +CPU= 34.757
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001728
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001728
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001632
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001644
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001656
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001668
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001680
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001692
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001704
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001716
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001716
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001716
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307_17.zip
Uploading the intermediate file: upload_file_17.zip
  03:48:54 STEP 1729 H= 432:15 +CPU= 24.208
  03:49:18 STEP 1730 H= 432:30 +CPU= 24.136
  03:49:42 STEP 1731 H= 432:45 +CPU= 23.881
  03:50:08 STEP 1732 H= 433:00 +CPU= 25.287
  03:50:32 STEP 1733 H= 433:15 +CPU= 24.068
  03:50:57 STEP 1734 H= 433:30 +CPU= 25.047
  03:51:22 STEP 1735 H= 433:45 +CPU= 24.688
  03:51:47 STEP 1736 H= 434:00 +CPU= 25.774
  03:52:12 STEP 1737 H= 434:15 +CPU= 24.659
  03:52:37 STEP 1738 H= 434:30 +CPU= 25.053
  03:53:03 STEP 1739 H= 434:45 +CPU= 25.322
  03:53:34 STEP 1740 H= 435:00 +CPU= 31.361
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001740
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001740
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001740
  03:53:59 STEP 1741 H= 435:15 +CPU= 25.134
  03:54:24 STEP 1742 H= 435:30 +CPU= 24.781
  03:54:48 STEP 1743 H= 435:45 +CPU= 23.681
  03:55:14 STEP 1744 H= 436:00 +CPU= 25.639
  03:55:40 STEP 1745 H= 436:15 +CPU= 26.033
  03:56:05 STEP 1746 H= 436:30 +CPU= 25.158
  03:56:29 STEP 1747 H= 436:45 +CPU= 24.384
  03:56:55 STEP 1748 H= 437:00 +CPU= 25.474
  03:57:19 STEP 1749 H= 437:15 +CPU= 24.473
  03:57:44 STEP 1750 H= 437:30 +CPU= 25.112
  03:58:10 STEP 1751 H= 437:45 +CPU= 25.918
  03:58:41 STEP 1752 H= 438:00 +CPU= 31.006
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001752
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001752
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001752
  03:59:07 STEP 1753 H= 438:15 +CPU= 25.223
  03:59:31 STEP 1754 H= 438:30 +CPU= 24.319
  03:59:54 STEP 1755 H= 438:45 +CPU= 22.626
  04:00:12 STEP 1756 H= 439:00 +CPU= 18.610
  04:00:30 STEP 1757 H= 439:15 +CPU= 18.125
  04:00:53 STEP 1758 H= 439:30 +CPU= 22.852
  04:01:18 STEP 1759 H= 439:45 +CPU= 25.036
  04:01:44 STEP 1760 H= 440:00 +CPU= 25.700
  04:02:09 STEP 1761 H= 440:15 +CPU= 24.983
  04:02:34 STEP 1762 H= 440:30 +CPU= 24.351
  04:02:58 STEP 1763 H= 440:45 +CPU= 24.581
  04:03:30 STEP 1764 H= 441:00 +CPU= 31.348
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001764
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001764
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001764
  04:03:54 STEP 1765 H= 441:15 +CPU= 23.952
  04:04:18 STEP 1766 H= 441:30 +CPU= 24.227
  04:04:44 STEP 1767 H= 441:45 +CPU= 25.615
  04:05:10 STEP 1768 H= 442:00 +CPU= 26.103
  04:05:35 STEP 1769 H= 442:15 +CPU= 25.427
  04:05:58 STEP 1770 H= 442:30 +CPU= 22.975
  04:06:23 STEP 1771 H= 442:45 +CPU= 25.082
  04:06:49 STEP 1772 H= 443:00 +CPU= 25.962
  04:07:14 STEP 1773 H= 443:15 +CPU= 24.617
  04:07:39 STEP 1774 H= 443:30 +CPU= 24.977
  04:08:04 STEP 1775 H= 443:45 +CPU= 24.514
  04:08:38 STEP 1776 H= 444:00 +CPU= 33.941
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001776
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001776
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001776
  04:09:02 STEP 1777 H= 444:15 +CPU= 24.317
  04:09:27 STEP 1778 H= 444:30 +CPU= 24.983
  04:09:52 STEP 1779 H= 444:45 +CPU= 24.982
  04:10:17 STEP 1780 H= 445:00 +CPU= 24.812
  04:10:41 STEP 1781 H= 445:15 +CPU= 24.151
  04:11:06 STEP 1782 H= 445:30 +CPU= 24.780
  04:11:31 STEP 1783 H= 445:45 +CPU= 24.657
  04:11:56 STEP 1784 H= 446:00 +CPU= 24.978
  04:12:20 STEP 1785 H= 446:15 +CPU= 24.110
  04:12:44 STEP 1786 H= 446:30 +CPU= 24.362
  04:13:09 STEP 1787 H= 446:45 +CPU= 25.065
  04:13:38 STEP 1788 H= 447:00 +CPU= 28.343
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001788
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001788
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001788
  04:13:59 STEP 1789 H= 447:15 +CPU= 21.603
  04:14:22 STEP 1790 H= 447:30 +CPU= 22.593
  04:14:45 STEP 1791 H= 447:45 +CPU= 23.352
  04:15:08 STEP 1792 H= 448:00 +CPU= 22.879
  04:15:31 STEP 1793 H= 448:15 +CPU= 22.550
  04:15:53 STEP 1794 H= 448:30 +CPU= 21.828
  04:16:16 STEP 1795 H= 448:45 +CPU= 23.287
  04:16:40 STEP 1796 H= 449:00 +CPU= 23.926
  04:17:03 STEP 1797 H= 449:15 +CPU= 22.644
  04:17:26 STEP 1798 H= 449:30 +CPU= 23.324
  04:17:50 STEP 1799 H= 449:45 +CPU= 23.884
  04:18:21 STEP 1800 H= 450:00 +CPU= 30.565
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001800
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001800
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001800
  04:18:40 STEP 1801 H= 450:15 +CPU= 19.405
  04:18:57 STEP 1802 H= 450:30 +CPU= 16.586
  04:19:15 STEP 1803 H= 450:45 +CPU= 18.199
  04:19:40 STEP 1804 H= 451:00 +CPU= 24.771
  04:20:04 STEP 1805 H= 451:15 +CPU= 23.899
  04:20:28 STEP 1806 H= 451:30 +CPU= 24.008
  04:20:52 STEP 1807 H= 451:45 +CPU= 24.078
  04:21:18 STEP 1808 H= 452:00 +CPU= 25.706
  04:21:41 STEP 1809 H= 452:15 +CPU= 23.599
  04:22:05 STEP 1810 H= 452:30 +CPU= 23.769
  04:22:29 STEP 1811 H= 452:45 +CPU= 23.808
  04:22:59 STEP 1812 H= 453:00 +CPU= 29.909
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001812
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001812
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001812
  04:23:22 STEP 1813 H= 453:15 +CPU= 23.080
  04:23:46 STEP 1814 H= 453:30 +CPU= 23.667
  04:24:11 STEP 1815 H= 453:45 +CPU= 25.166
  04:24:36 STEP 1816 H= 454:00 +CPU= 24.842
  04:25:01 STEP 1817 H= 454:15 +CPU= 24.857
  04:25:24 STEP 1818 H= 454:30 +CPU= 23.257
  04:25:47 STEP 1819 H= 454:45 +CPU= 23.105
  04:26:12 STEP 1820 H= 455:00 +CPU= 24.488
  04:26:36 STEP 1821 H= 455:15 +CPU= 24.636
  04:27:01 STEP 1822 H= 455:30 +CPU= 24.224
  04:27:25 STEP 1823 H= 455:45 +CPU= 24.357
  04:27:58 STEP 1824 H= 456:00 +CPU= 33.053
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001824
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001824
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001728
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001740
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001752
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001764
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001776
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001788
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001800
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001812
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001812
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001812
Zipping up the intermediate file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307_18.zip
Uploading the intermediate file: upload_file_18.zip
  04:28:22 STEP 1825 H= 456:15 +CPU= 23.762
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3tv
(argv4) batchid: 1018
(argv5) wuid: 12293290
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /boinc_data/boinc/slots/7
Project directory is: /boinc_data/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290
..mkdir for temp folder for results failed
Copying: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /boinc_data/boinc/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /boinc_data/boinc/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ee46bccdd5523a26a1e57cd692e52d0d to: /boinc_data/boinc/slots/7/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290.zip
Unzipping the namelist zip file: /boinc_data/boinc/slots/7/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290.zip
ic_ancil_file: ic_ancil_12293290
ifsdata_file: ifsdata_12293290
climate_data_file: clim_data_12293290
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /boinc_data/boinc/slots/7/ic_ancil_12293290.zip
Unzipping the IC ancils zip file: /boinc_data/boinc/slots/7/ic_ancil_12293290.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_09119fb5b6f9a2521551e0fe321917de to: /boinc_data/boinc/slots/7/ifsdata/ifsdata_12293290.zip
Unzipping the ifsdata_zip file: /boinc_data/boinc/slots/7/ifsdata/ifsdata_12293290.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /boinc_data/boinc/slots/7/159l_2/clim_data_12293290.zip
Unzipping the climate data zip file: /boinc_data/boinc/slots/7/159l_2/clim_data_12293290.zip
Checking for progress XML file: /boinc_data/boinc/slots/7/progress_file_12293290.xml
Opened progress file ok : /boinc_data/boinc/slots/7/progress_file_12293290.xml
-- Model is restarting --
Adjusting last_iter, 1825, to previous model restart step.
Creating progress file: /boinc_data/boinc/slots/7/progress_file_12293290.xml
last_cpu_time: 50655
upload_file_number: 19
last_iter: 1825
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307
The child process has been launched with process id: 874159
Executing the command: /boinc_data/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:530FCTest:1:1:874159:874159] [20240616:043251:1718530371:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:33:19 STEP 1824 H= 456:00 +CPU= 24.160
  04:33:42 STEP 1825 H= 456:15 +CPU= 23.283
  04:34:06 STEP 1826 H= 456:30 +CPU= 23.982
  04:34:29 STEP 1827 H= 456:45 +CPU= 22.965
  04:34:54 STEP 1828 H= 457:00 +CPU= 24.205
  04:35:19 STEP 1829 H= 457:15 +CPU= 24.902
  04:35:42 STEP 1830 H= 457:30 +CPU= 23.514
  04:36:06 STEP 1831 H= 457:45 +CPU= 23.314
  04:36:30 STEP 1832 H= 458:00 +CPU= 24.163
  04:36:54 STEP 1833 H= 458:15 +CPU= 23.938
  04:37:17 STEP 1834 H= 458:30 +CPU= 23.202
  04:37:40 STEP 1835 H= 458:45 +CPU= 23.370
  04:38:10 STEP 1836 H= 459:00 +CPU= 29.664
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001836
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001836
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001836
  04:38:33 STEP 1837 H= 459:15 +CPU= 22.973
  04:38:57 STEP 1838 H= 459:30 +CPU= 23.390
  04:39:20 STEP 1839 H= 459:45 +CPU= 23.518
  04:39:45 STEP 1840 H= 460:00 +CPU= 24.591
  04:40:07 STEP 1841 H= 460:15 +CPU= 22.203
  04:40:24 STEP 1842 H= 460:30 +CPU= 17.347
  04:40:42 STEP 1843 H= 460:45 +CPU= 17.458
  04:41:02 STEP 1844 H= 461:00 +CPU= 20.038
  04:41:26 STEP 1845 H= 461:15 +CPU= 24.556
  04:41:50 STEP 1846 H= 461:30 +CPU= 23.277
  04:42:13 STEP 1847 H= 461:45 +CPU= 22.830
  04:42:42 STEP 1848 H= 462:00 +CPU= 29.190
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001848
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001848
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001848
  04:43:06 STEP 1849 H= 462:15 +CPU= 23.750
  04:43:31 STEP 1850 H= 462:30 +CPU= 25.048
  04:43:56 STEP 1851 H= 462:45 +CPU= 25.388
  04:44:21 STEP 1852 H= 463:00 +CPU= 24.785
  04:44:47 STEP 1853 H= 463:15 +CPU= 26.269
  04:45:11 STEP 1854 H= 463:30 +CPU= 24.045
  04:45:36 STEP 1855 H= 463:45 +CPU= 24.140
  04:46:01 STEP 1856 H= 464:00 +CPU= 25.292
  04:46:25 STEP 1857 H= 464:15 +CPU= 24.461
  04:46:50 STEP 1858 H= 464:30 +CPU= 24.362
  04:47:14 STEP 1859 H= 464:45 +CPU= 24.189
  04:47:45 STEP 1860 H= 465:00 +CPU= 31.275
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001860
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001860
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001860
  04:48:10 STEP 1861 H= 465:15 +CPU= 24.517
  04:48:33 STEP 1862 H= 465:30 +CPU= 23.375
  04:48:57 STEP 1863 H= 465:45 +CPU= 24.069
  04:49:20 STEP 1864 H= 466:00 +CPU= 23.073
  04:49:44 STEP 1865 H= 466:15 +CPU= 23.924
  04:50:08 STEP 1866 H= 466:30 +CPU= 23.603
  04:50:31 STEP 1867 H= 466:45 +CPU= 22.696
  04:50:54 STEP 1868 H= 467:00 +CPU= 23.525
  04:51:19 STEP 1869 H= 467:15 +CPU= 24.409
  04:51:43 STEP 1870 H= 467:30 +CPU= 23.879
  04:52:07 STEP 1871 H= 467:45 +CPU= 24.221
  04:52:41 STEP 1872 H= 468:00 +CPU= 33.757
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001872
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001872
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001872
  04:53:04 STEP 1873 H= 468:15 +CPU= 23.353
  04:53:28 STEP 1874 H= 468:30 +CPU= 23.743
  04:53:51 STEP 1875 H= 468:45 +CPU= 23.313
  04:54:15 STEP 1876 H= 469:00 +CPU= 23.924
  04:54:39 STEP 1877 H= 469:15 +CPU= 23.361
  04:55:04 STEP 1878 H= 469:30 +CPU= 24.902
  04:55:28 STEP 1879 H= 469:45 +CPU= 24.304
  04:55:51 STEP 1880 H= 470:00 +CPU= 22.959
  04:56:14 STEP 1881 H= 470:15 +CPU= 22.810
  04:56:37 STEP 1882 H= 470:30 +CPU= 23.398
  04:57:01 STEP 1883 H= 470:45 +CPU= 23.756
  04:57:32 STEP 1884 H= 471:00 +CPU= 30.594
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001884
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001884
  04:57:56 STEP 1885 H= 471:15 +CPU= 23.775
  04:58:19 STEP 1886 H= 471:30 +CPU= 23.658
  04:58:43 STEP 1887 H= 471:45 +CPU= 23.207
  04:59:06 STEP 1888 H= 472:00 +CPU= 23.559
  04:59:28 STEP 1889 H= 472:15 +CPU= 21.560
  04:59:45 STEP 1890 H= 472:30 +CPU= 17.483
  05:00:03 STEP 1891 H= 472:45 +CPU= 18.165
  05:00:27 STEP 1892 H= 473:00 +CPU= 23.994
  05:00:51 STEP 1893 H= 473:15 +CPU= 23.648
  05:01:15 STEP 1894 H= 473:30 +CPU= 23.658
  05:01:39 STEP 1895 H= 473:45 +CPU= 23.982
  05:02:10 STEP 1896 H= 474:00 +CPU= 30.846
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001896
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001896
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001896
  05:02:34 STEP 1897 H= 474:15 +CPU= 24.254
  05:02:59 STEP 1898 H= 474:30 +CPU= 24.950
  05:03:24 STEP 1899 H= 474:45 +CPU= 24.795
  05:03:50 STEP 1900 H= 475:00 +CPU= 25.941
  05:04:14 STEP 1901 H= 475:15 +CPU= 24.232
  05:04:40 STEP 1902 H= 475:30 +CPU= 25.458
  05:05:04 STEP 1903 H= 475:45 +CPU= 24.566
  05:05:30 STEP 1904 H= 476:00 +CPU= 25.594
  05:05:55 STEP 1905 H= 476:15 +CPU= 24.708
  05:06:19 STEP 1906 H= 476:30 +CPU= 23.959
  05:06:44 STEP 1907 H= 476:45 +CPU= 25.323
  05:07:14 STEP 1908 H= 477:00 +CPU= 29.922
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001908
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001908
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001908
  05:07:38 STEP 1909 H= 477:15 +CPU= 24.195
  05:08:03 STEP 1910 H= 477:30 +CPU= 24.670
  05:08:28 STEP 1911 H= 477:45 +CPU= 24.842
  05:08:54 STEP 1912 H= 478:00 +CPU= 25.665
  05:09:18 STEP 1913 H= 478:15 +CPU= 24.080
  05:09:43 STEP 1914 H= 478:30 +CPU= 24.999
  05:10:07 STEP 1915 H= 478:45 +CPU= 24.481
  05:10:32 STEP 1916 H= 479:00 +CPU= 24.417
  05:10:56 STEP 1917 H= 479:15 +CPU= 24.173
  05:11:21 STEP 1918 H= 479:30 +CPU= 24.429
  05:11:45 STEP 1919 H= 479:45 +CPU= 24.441
  05:12:15 STEP 1920 H= 480:00 +CPU= 29.581
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1350775517E-20      I       0.1846739203E-07      I
ICloud fractionI      0.0000000000E+00      I       0.3061893483E-01      I
I  Relat. Hum. I      0.1318340836E+00      I       0.1824197847E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1218172350E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    397061316 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS794122.6 Mbytes/s, TOTAL TIME=    0.0067( 7.5%)
 05:12:15 STEP 1920 H= 480:00 +CPU= 29.581
 END CNT3
 NSTEP =  1920 CNT0   000000000  20.139  20.139  20.157   0.999               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         2860/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       46      1044068     2860 ************     6.2     1.5     4.7     2.3     9.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 2363523.9 2363523.9    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 2359803.2 2359803.2     99.84      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    4205.4    4205.4     18.68      0.00
   9 SPCM     - SPECTRAL COMP.                      96     555.6     555.6      2.26      0.00
  10 SCAN2M - PHYSICS                               97   10953.9   10953.9     44.96      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     845.4     845.4      0.29      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      37.0      37.0      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      23.3      23.3      0.01      0.00
  27 RESTART FILES - WRITING                         2    3381.4    3381.4      0.29      0.00
  28 RESTART FILES - READING                         1    2971.9    2971.9      0.13      0.00
  29 SU4FPOS IN CNT4                                 8       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    5513.5    5513.5      1.87      0.00
  31 POSDDH IN STEPO                                 1       0.7       0.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    5496.1    5496.1     22.56      0.00
  39 SU0YOMB                                         1     487.2     487.2      0.02      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      18.2      18.2      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     269.5     269.5      1.09      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      48.2      48.2      0.20      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     356.4     356.4      3.20      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     217.8     217.8      1.84      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      81.3      81.3      0.69      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     105.3     105.3      0.94      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      52.2      52.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      57.2      57.2      0.51      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      31.2      31.2      0.26      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212      43.4      43.4      0.39      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      24.6      24.6      0.21      0.00
 400 GSTATS                                      12215       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11311       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     2363.5 CPU TIME    2359.4 VECTOR TIME     2359.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      732.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /boinc_data/boinc/slots/7/ICMGGh7zg+001920
Moving to projects directory: /boinc_data/boinc/slots/7/ICMSHh7zg+001920
Moving to projects directory: /boinc_data/boinc/slots/7/ICMUAh7zg+001920
Adding to the zip: /boinc_data/boinc/slots/7/NODE.001_01
Adding to the zip: /boinc_data/boinc/slots/7/ifs.stat
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001860
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001860
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001884
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001824
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001908
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001836
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001872
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001896
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMGGh7zg+001920
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMSHh7zg+001848
Adding to the zip: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_12293290/ICMUAh7zg+001860
Zipping up the final file: /boinc_data/boinc/projects/climateprediction.net/oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0_r245801307_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
05:15:47 (874155): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 Jun 2024 10:36:17 1496046 22446902 oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0 1,727,100 53,014 0.0307
16 Jun 2024 02:50:11 1496046 22446902 oifs_43r3_bl_a3tv_2016092300_20_1018_12293290_0 864,000 26,801 0.0310


©2024 climateprediction.net