climateprediction.net home page
Task 22266093

Task 22266093

Name oifs_43r3_ps_0435_1995050100_123_964_12181079_0
Workunit 12181079
Created 22 Dec 2022, 8:21:15 UTC
Sent 22 Dec 2022, 8:33:16 UTC
Report deadline 21 Jan 2023, 8:33:16 UTC
Received 30 Dec 2022, 8:47:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1489681
Run time 12 hours 12 min 47 sec
CPU time 11 hours 34 min 46 sec
Validate state Invalid
Credit 8,588.16
Device peak FLOPS 6.59 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,763.59 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,168.29 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
NC client, suspending the child process
Resuming the child process
  21:30:19 STEP 1636 H=1636:00 +CPU= 16.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:30:37 STEP 1637 H=1637:00 +CPU= 16.458
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:31:09 STEP 1638 H=1638:00 +CPU= 27.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:31:28 STEP 1639 H=1639:00 +CPU= 16.917
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:31:47 STEP 1640 H=1640:00 +CPU= 17.380
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:32:17 STEP 1641 H=1641:00 +CPU= 27.258
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:32:36 STEP 1642 H=1642:00 +CPU= 17.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:32:55 STEP 1643 H=1643:00 +CPU= 16.911
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:33:29 STEP 1644 H=1644:00 +CPU= 30.357
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001644
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001644
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001644
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:33:48 STEP 1645 H=1645:00 +CPU= 16.917
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:34:08 STEP 1646 H=1646:00 +CPU= 17.357
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:34:37 STEP 1647 H=1647:00 +CPU= 27.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:34:56 STEP 1648 H=1648:00 +CPU= 17.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:35:15 STEP 1649 H=1649:00 +CPU= 16.888
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:35:47 STEP 1650 H=1650:00 +CPU= 27.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:36:06 STEP 1651 H=1651:00 +CPU= 16.872
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:36:25 STEP 1652 H=1652:00 +CPU= 17.246
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:36:49 STEP 1653 H=1653:00 +CPU= 21.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:37:06 STEP 1654 H=1654:00 +CPU= 14.916
  21:37:23 STEP 1655 H=1655:00 +CPU= 16.899
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:37:57 STEP 1656 H=1656:00 +CPU= 30.589
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001656
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001656
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001632
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001632
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001632
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001644
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001644
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001644
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_68.zip
Uploading the intermediate file: upload_file_68.zip
  21:38:15 STEP 1657 H=1657:00 +CPU= 14.491
  21:38:30 STEP 1658 H=1658:00 +CPU= 15.355
  21:38:56 STEP 1659 H=1659:00 +CPU= 26.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:39:15 STEP 1660 H=1660:00 +CPU= 16.888
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:39:34 STEP 1661 H=1661:00 +CPU= 16.827
  21:40:02 STEP 1662 H=1662:00 +CPU= 27.725
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:40:20 STEP 1663 H=1663:00 +CPU= 16.236
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:40:39 STEP 1664 H=1664:00 +CPU= 17.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:41:10 STEP 1665 H=1665:00 +CPU= 27.194
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:41:30 STEP 1666 H=1666:00 +CPU= 17.328
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:41:49 STEP 1667 H=1667:00 +CPU= 16.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:42:21 STEP 1668 H=1668:00 +CPU= 30.334
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001668
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001668
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001668
  21:42:40 STEP 1669 H=1669:00 +CPU= 16.847
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:42:59 STEP 1670 H=1670:00 +CPU= 17.320
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:43:31 STEP 1671 H=1671:00 +CPU= 27.159
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:43:50 STEP 1672 H=1672:00 +CPU= 17.149
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:44:08 STEP 1673 H=1673:00 +CPU= 16.416
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:44:37 STEP 1674 H=1674:00 +CPU= 26.998
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:44:56 STEP 1675 H=1675:00 +CPU= 16.387
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:45:14 STEP 1676 H=1676:00 +CPU= 16.864
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:45:45 STEP 1677 H=1677:00 +CPU= 26.465
  21:46:02 STEP 1678 H=1678:00 +CPU= 16.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:46:21 STEP 1679 H=1679:00 +CPU= 17.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:46:57 STEP 1680 H=1680:00 +CPU= 31.644
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001680
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001680
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001656
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001668
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001668
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001668
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_69.zip
Uploading the intermediate file: upload_file_69.zip
  21:47:16 STEP 1681 H=1681:00 +CPU= 16.914
  21:47:33 STEP 1682 H=1682:00 +CPU= 17.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:48:02 STEP 1683 H=1683:00 +CPU= 27.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:48:22 STEP 1684 H=1684:00 +CPU= 17.408
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:48:40 STEP 1685 H=1685:00 +CPU= 16.515
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:49:12 STEP 1686 H=1686:00 +CPU= 27.864
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:49:31 STEP 1687 H=1687:00 +CPU= 16.954
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:49:51 STEP 1688 H=1688:00 +CPU= 17.400
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:50:20 STEP 1689 H=1689:00 +CPU= 27.355
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:50:39 STEP 1690 H=1690:00 +CPU= 17.383
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:50:58 STEP 1691 H=1691:00 +CPU= 16.840
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:51:33 STEP 1692 H=1692:00 +CPU= 30.460
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001692
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001692
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:51:52 STEP 1693 H=1693:00 +CPU= 16.944
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:52:11 STEP 1694 H=1694:00 +CPU= 17.439
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:52:41 STEP 1695 H=1695:00 +CPU= 27.346
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:53:00 STEP 1696 H=1696:00 +CPU= 17.394
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:53:19 STEP 1697 H=1697:00 +CPU= 16.946
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:53:51 STEP 1698 H=1698:00 +CPU= 27.767
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:54:10 STEP 1699 H=1699:00 +CPU= 16.924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:54:29 STEP 1700 H=1700:00 +CPU= 17.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:54:59 STEP 1701 H=1701:00 +CPU= 27.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:55:18 STEP 1702 H=1702:00 +CPU= 17.358
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:55:37 STEP 1703 H=1703:00 +CPU= 17.354
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:56:13 STEP 1704 H=1704:00 +CPU= 31.471
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001704
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001704
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001680
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001692
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001692
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001692
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_70.zip
Uploading the intermediate file: upload_file_70.zip
  21:56:30 STEP 1705 H=1705:00 +CPU= 16.775
Uploading trickle at timestep: 6134400
  21:56:47 STEP 1706 H=1706:00 +CPU= 17.182
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:57:16 STEP 1707 H=1707:00 +CPU= 27.126
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:57:35 STEP 1708 H=1708:00 +CPU= 17.284
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:57:54 STEP 1709 H=1709:00 +CPU= 16.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:58:26 STEP 1710 H=1710:00 +CPU= 27.719
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:58:45 STEP 1711 H=1711:00 +CPU= 16.840
  21:59:02 STEP 1712 H=1712:00 +CPU= 17.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:59:33 STEP 1713 H=1713:00 +CPU= 27.157
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:59:53 STEP 1714 H=1714:00 +CPU= 17.296
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0435
(argv4) batchid: 964
(argv5) wuid: 12181079
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/nvme/scratch1/boinc/slots/1
Project directory is: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079
..mkdir for temp folder for results failed
Copying: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e2bfb140cd8aa1e91ea7e967da64f967 to: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_0435_1995050100_123_964_12181079.zip
Unzipping the namelist zip file: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_0435_1995050100_123_964_12181079.zip
ic_ancil_file: ic_ancil_12181079
ifsdata_file: ifsdata_12181079
climate_data_file: clim_data_12181079
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6daf3f584971cf80a09bf127d96865c7 to: /mnt/nvme/scratch1/boinc/slots/1/ic_ancil_12181079.zip
Unzipping the IC ancils zip file: /mnt/nvme/scratch1/boinc/slots/1/ic_ancil_12181079.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_577a197af9d8e1bfbd56e77858179c57 to: /mnt/nvme/scratch1/boinc/slots/1/ifsdata/ifsdata_12181079.zip
Unzipping the ifsdata_zip file: /mnt/nvme/scratch1/boinc/slots/1/ifsdata/ifsdata_12181079.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/nvme/scratch1/boinc/slots/1/159l_2/clim_data_12181079.zip
Unzipping the climate data zip file: /mnt/nvme/scratch1/boinc/slots/1/159l_2/clim_data_12181079.zip
Checking for progress XML file: /mnt/nvme/scratch1/boinc/slots/1/progress_file_12181079.xml
Opened progress file ok : /mnt/nvme/scratch1/boinc/slots/1/progress_file_12181079.xml
-- Model is restarting --
Adjusting last_iter, 1715, to previous model restart step.
Creating progress file: /mnt/nvme/scratch1/boinc/slots/1/progress_file_12181079.xml
last_cpu_time: 37910
upload_file_number: 71
last_iter: 1703
last_upload: 6134400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097
The child process has been launched with process id: 23373
Executing the command: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:00:35 STEP 1704 H=1704:00 +CPU= 27.447
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:00:51 STEP 1705 H=1705:00 +CPU= 13.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:01:09 STEP 1706 H=1706:00 +CPU= 16.356
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:01:39 STEP 1707 H=1707:00 +CPU= 27.664
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:01:59 STEP 1708 H=1708:00 +CPU= 17.846
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:02:18 STEP 1709 H=1709:00 +CPU= 17.416
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:02:50 STEP 1710 H=1710:00 +CPU= 28.019
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:03:09 STEP 1711 H=1711:00 +CPU= 16.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:03:29 STEP 1712 H=1712:00 +CPU= 17.413
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:03:55 STEP 1713 H=1713:00 +CPU= 24.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:04:12 STEP 1714 H=1714:00 +CPU= 14.171
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:04:27 STEP 1715 H=1715:00 +CPU= 13.839
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:04:59 STEP 1716 H=1716:00 +CPU= 29.192
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001716
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001716
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001716
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:05:18 STEP 1717 H=1717:00 +CPU= 16.902
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:05:35 STEP 1718 H=1718:00 +CPU= 14.956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:06:02 STEP 1719 H=1719:00 +CPU= 25.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:06:21 STEP 1720 H=1720:00 +CPU= 16.619
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:06:39 STEP 1721 H=1721:00 +CPU= 16.775
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:07:11 STEP 1722 H=1722:00 +CPU= 27.688
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:07:30 STEP 1723 H=1723:00 +CPU= 16.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:07:49 STEP 1724 H=1724:00 +CPU= 17.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:08:19 STEP 1725 H=1725:00 +CPU= 27.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:08:39 STEP 1726 H=1726:00 +CPU= 17.776
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:08:58 STEP 1727 H=1727:00 +CPU= 17.752
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:09:35 STEP 1728 H=1728:00 +CPU= 32.410
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001728
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001728
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001704
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001716
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001716
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001716
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_71.zip
Uploading the intermediate file: upload_file_71.zip
  08:09:52 STEP 1729 H=1729:00 +CPU= 17.171
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:10:12 STEP 1730 H=1730:00 +CPU= 17.837
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:10:42 STEP 1731 H=1731:00 +CPU= 27.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:11:01 STEP 1732 H=1732:00 +CPU= 17.381
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:11:20 STEP 1733 H=1733:00 +CPU= 17.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:11:53 STEP 1734 H=1734:00 +CPU= 28.456
  08:12:10 STEP 1735 H=1735:00 +CPU= 17.239
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:12:30 STEP 1736 H=1736:00 +CPU= 17.919
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:13:00 STEP 1737 H=1737:00 +CPU= 27.851
  08:13:18 STEP 1738 H=1738:00 +CPU= 17.834
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:13:37 STEP 1739 H=1739:00 +CPU= 17.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:14:12 STEP 1740 H=1740:00 +CPU= 30.923
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001740
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001740
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:14:32 STEP 1741 H=1741:00 +CPU= 17.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:14:52 STEP 1742 H=1742:00 +CPU= 17.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:15:22 STEP 1743 H=1743:00 +CPU= 27.869
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:15:41 STEP 1744 H=1744:00 +CPU= 17.787
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:16:01 STEP 1745 H=1745:00 +CPU= 17.230
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:16:33 STEP 1746 H=1746:00 +CPU= 28.218
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:16:52 STEP 1747 H=1747:00 +CPU= 17.110
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:17:12 STEP 1748 H=1748:00 +CPU= 17.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:17:41 STEP 1749 H=1749:00 +CPU= 27.397
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:18:01 STEP 1750 H=1750:00 +CPU= 17.831
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:18:21 STEP 1751 H=1751:00 +CPU= 17.457
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:18:57 STEP 1752 H=1752:00 +CPU= 32.091
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001752
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001752
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001728
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001740
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001740
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001740
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_72.zip
Uploading the intermediate file: upload_file_72.zip
  08:19:14 STEP 1753 H=1753:00 +CPU= 17.352
  08:19:32 STEP 1754 H=1754:00 +CPU= 17.792
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:20:02 STEP 1755 H=1755:00 +CPU= 27.884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:20:22 STEP 1756 H=1756:00 +CPU= 17.719
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:20:41 STEP 1757 H=1757:00 +CPU= 17.267
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:21:13 STEP 1758 H=1758:00 +CPU= 27.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:21:32 STEP 1759 H=1759:00 +CPU= 17.512
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:21:52 STEP 1760 H=1760:00 +CPU= 17.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:22:24 STEP 1761 H=1761:00 +CPU= 28.116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:22:45 STEP 1762 H=1762:00 +CPU= 18.142
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:23:04 STEP 1763 H=1763:00 +CPU= 17.667
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:23:38 STEP 1764 H=1764:00 +CPU= 31.173
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001764
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001764
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001764
  08:23:57 STEP 1765 H=1765:00 +CPU= 17.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:24:17 STEP 1766 H=1766:00 +CPU= 17.967
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:24:49 STEP 1767 H=1767:00 +CPU= 28.066
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:25:08 STEP 1768 H=1768:00 +CPU= 16.903
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:25:27 STEP 1769 H=1769:00 +CPU= 17.111
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:25:58 STEP 1770 H=1770:00 +CPU= 28.632
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:26:17 STEP 1771 H=1771:00 +CPU= 17.206
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:26:37 STEP 1772 H=1772:00 +CPU= 17.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:27:08 STEP 1773 H=1773:00 +CPU= 26.915
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:27:28 STEP 1774 H=1774:00 +CPU= 17.845
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:27:47 STEP 1775 H=1775:00 +CPU= 17.714
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:28:21 STEP 1776 H=1776:00 +CPU= 31.650
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001776
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001776
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001752
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001764
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001764
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001764
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_73.zip
Uploading the intermediate file: upload_file_73.zip
  08:28:40 STEP 1777 H=1777:00 +CPU= 17.341
  08:28:58 STEP 1778 H=1778:00 +CPU= 17.891
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:29:30 STEP 1779 H=1779:00 +CPU= 27.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:29:50 STEP 1780 H=1780:00 +CPU= 17.857
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:30:10 STEP 1781 H=1781:00 +CPU= 17.223
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:30:40 STEP 1782 H=1782:00 +CPU= 28.541
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:31:00 STEP 1783 H=1783:00 +CPU= 17.563
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:31:20 STEP 1784 H=1784:00 +CPU= 18.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:31:52 STEP 1785 H=1785:00 +CPU= 27.911
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:32:12 STEP 1786 H=1786:00 +CPU= 17.676
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:32:32 STEP 1787 H=1787:00 +CPU= 17.465
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:33:07 STEP 1788 H=1788:00 +CPU= 31.230
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001788
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001788
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:33:26 STEP 1789 H=1789:00 +CPU= 17.441
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:33:46 STEP 1790 H=1790:00 +CPU= 17.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:34:16 STEP 1791 H=1791:00 +CPU= 28.216
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:34:37 STEP 1792 H=1792:00 +CPU= 18.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:34:56 STEP 1793 H=1793:00 +CPU= 17.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:35:29 STEP 1794 H=1794:00 +CPU= 28.631
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:35:48 STEP 1795 H=1795:00 +CPU= 17.444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:36:08 STEP 1796 H=1796:00 +CPU= 17.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:36:38 STEP 1797 H=1797:00 +CPU= 27.549
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:36:57 STEP 1798 H=1798:00 +CPU= 17.518
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:37:17 STEP 1799 H=1799:00 +CPU= 17.534
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:37:54 STEP 1800 H=1800:00 +CPU= 32.744
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001800
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001800
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001776
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001788
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001788
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001788
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_74.zip
Uploading the intermediate file: upload_file_74.zip
  08:38:11 STEP 1801 H=1801:00 +CPU= 17.477
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:38:31 STEP 1802 H=1802:00 +CPU= 17.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:39:03 STEP 1803 H=1803:00 +CPU= 28.182
  08:39:21 STEP 1804 H=1804:00 +CPU= 17.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:39:40 STEP 1805 H=1805:00 +CPU= 17.220
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:40:12 STEP 1806 H=1806:00 +CPU= 27.931
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:40:31 STEP 1807 H=1807:00 +CPU= 17.221
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:40:51 STEP 1808 H=1808:00 +CPU= 17.651
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:41:18 STEP 1809 H=1809:00 +CPU= 25.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:41:37 STEP 1810 H=1810:00 +CPU= 16.699
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:41:56 STEP 1811 H=1811:00 +CPU= 17.219
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:42:31 STEP 1812 H=1812:00 +CPU= 30.749
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001812
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001812
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:42:51 STEP 1813 H=1813:00 +CPU= 17.560
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:43:11 STEP 1814 H=1814:00 +CPU= 18.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:43:41 STEP 1815 H=1815:00 +CPU= 28.172
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:44:01 STEP 1816 H=1816:00 +CPU= 18.033
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:44:21 STEP 1817 H=1817:00 +CPU= 17.534
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:44:53 STEP 1818 H=1818:00 +CPU= 28.500
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:45:13 STEP 1819 H=1819:00 +CPU= 17.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:45:33 STEP 1820 H=1820:00 +CPU= 17.966
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:46:05 STEP 1821 H=1821:00 +CPU= 28.021
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:46:25 STEP 1822 H=1822:00 +CPU= 17.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:46:45 STEP 1823 H=1823:00 +CPU= 17.917
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:47:20 STEP 1824 H=1824:00 +CPU= 32.887
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001824
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001824
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001800
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001812
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001812
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001812
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_75.zip
Uploading the intermediate file: upload_file_75.zip
  08:47:37 STEP 1825 H=1825:00 +CPU= 17.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:47:57 STEP 1826 H=1826:00 +CPU= 18.112
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:48:29 STEP 1827 H=1827:00 +CPU= 28.082
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:48:50 STEP 1828 H=1828:00 +CPU= 18.099
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:49:09 STEP 1829 H=1829:00 +CPU= 17.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:49:40 STEP 1830 H=1830:00 +CPU= 28.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:49:59 STEP 1831 H=1831:00 +CPU= 17.445
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:50:20 STEP 1832 H=1832:00 +CPU= 17.945
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:50:52 STEP 1833 H=1833:00 +CPU= 28.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:51:12 STEP 1834 H=1834:00 +CPU= 17.994
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:51:32 STEP 1835 H=1835:00 +CPU= 17.439
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:52:07 STEP 1836 H=1836:00 +CPU= 31.237
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001836
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001836
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:52:27 STEP 1837 H=1837:00 +CPU= 17.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:52:46 STEP 1838 H=1838:00 +CPU= 17.849
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:53:17 STEP 1839 H=1839:00 +CPU= 28.056
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:53:36 STEP 1840 H=1840:00 +CPU= 17.958
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:53:56 STEP 1841 H=1841:00 +CPU= 17.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:54:28 STEP 1842 H=1842:00 +CPU= 27.864
  08:54:45 STEP 1843 H=1843:00 +CPU= 16.925
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:55:05 STEP 1844 H=1844:00 +CPU= 17.619
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:55:35 STEP 1845 H=1845:00 +CPU= 28.033
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:55:55 STEP 1846 H=1846:00 +CPU= 17.839
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:56:15 STEP 1847 H=1847:00 +CPU= 18.026
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:56:51 STEP 1848 H=1848:00 +CPU= 32.734
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001848
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001848
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001824
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001836
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001836
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001836
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_76.zip
Uploading the intermediate file: upload_file_76.zip
  08:57:09 STEP 1849 H=1849:00 +CPU= 17.358
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:57:29 STEP 1850 H=1850:00 +CPU= 17.938
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:57:59 STEP 1851 H=1851:00 +CPU= 28.042
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:58:19 STEP 1852 H=1852:00 +CPU= 17.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:58:39 STEP 1853 H=1853:00 +CPU= 17.584
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:59:11 STEP 1854 H=1854:00 +CPU= 28.680
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:59:31 STEP 1855 H=1855:00 +CPU= 17.715
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:59:52 STEP 1856 H=1856:00 +CPU= 18.252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:00:24 STEP 1857 H=1857:00 +CPU= 28.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:00:44 STEP 1858 H=1858:00 +CPU= 18.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:01:04 STEP 1859 H=1859:00 +CPU= 17.751
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:01:37 STEP 1860 H=1860:00 +CPU= 30.997
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001860
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001860
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001860
  09:01:56 STEP 1861 H=1861:00 +CPU= 17.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:02:17 STEP 1862 H=1862:00 +CPU= 18.146
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:02:49 STEP 1863 H=1863:00 +CPU= 28.241
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:03:09 STEP 1864 H=1864:00 +CPU= 18.028
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:03:29 STEP 1865 H=1865:00 +CPU= 17.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:03:59 STEP 1866 H=1866:00 +CPU= 28.784
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:04:18 STEP 1867 H=1867:00 +CPU= 16.655
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:04:38 STEP 1868 H=1868:00 +CPU= 18.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:05:11 STEP 1869 H=1869:00 +CPU= 28.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:05:31 STEP 1870 H=1870:00 +CPU= 18.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:05:51 STEP 1871 H=1871:00 +CPU= 18.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:06:28 STEP 1872 H=1872:00 +CPU= 33.041
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001872
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001872
Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001872
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001848
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001860
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001860
Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001860
Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_77.zip
Uploading the intermediate file: upload_file_77.zip
  09:06:45 STEP 1873 H=1873:00 +CPU= 17.421
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:07:06 STEP 1874 H=1874:00 +CPU= 18.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:07:36 STEP 1875 H=1875:00 +CPU= 28.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:07:55 STEP 1876 H=1876:00 +CPU= 17.632
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:08:15 STEP 1877 H=1877:00 +CPU= 17.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:08:48 STEP 1878 H=1878:00 +CPU= 28.866
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:09:08 STEP 1879 H=1879:00 +CPU= 17.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:09:29 STEP 1880 H=1880:00 +CPU= 18.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:02:04 1489681 22266093 oifs_43r3_ps_0435_1995050100_123_964_12181079_0 6,134,400 37,710 0.0061
09 May 2023 16:01:59 1489681 22266093 oifs_43r3_ps_0435_1995050100_123_964_12181079_0 5,270,400 32,729 0.0062
09 May 2023 16:01:59 1489681 22266093 oifs_43r3_ps_0435_1995050100_123_964_12181079_0 5,270,400 32,729 0.0062
09 May 2023 15:57:06 1489681 22266093 oifs_43r3_ps_0435_1995050100_123_964_12181079_0 1,728,000 10,583 0.0061
09 May 2023 15:57:01 1489681 22266093 oifs_43r3_ps_0435_1995050100_123_964_12181079_0 864,000 5,361 0.0062


©2024 cpdn.org