Name | oifs_43r3_ps_0435_1995050100_123_964_12181079_0 |
Workunit | 12181079 |
Created | 22 Dec 2022, 8:21:15 UTC |
Sent | 22 Dec 2022, 8:33:16 UTC |
Report deadline | 21 Jan 2023, 8:33:16 UTC |
Received | 30 Dec 2022, 8:47:44 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1489681 |
Run time | 12 hours 12 min 47 sec |
CPU time | 11 hours 34 min 46 sec |
Validate state | Invalid |
Credit | 8,588.16 |
Device peak FLOPS | 6.59 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,763.59 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 7,168.29 MB |
Stderr | <core_client_version>7.20.2</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> NC client, suspending the child process Resuming the child process 21:30:19 STEP 1636 H=1636:00 +CPU= 16.881 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:30:37 STEP 1637 H=1637:00 +CPU= 16.458 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:09 STEP 1638 H=1638:00 +CPU= 27.648 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:28 STEP 1639 H=1639:00 +CPU= 16.917 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:47 STEP 1640 H=1640:00 +CPU= 17.380 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:17 STEP 1641 H=1641:00 +CPU= 27.258 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:36 STEP 1642 H=1642:00 +CPU= 17.402 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:55 STEP 1643 H=1643:00 +CPU= 16.911 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:33:29 STEP 1644 H=1644:00 +CPU= 30.357 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001644 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001644 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001644 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:33:48 STEP 1645 H=1645:00 +CPU= 16.917 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:34:08 STEP 1646 H=1646:00 +CPU= 17.357 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:34:37 STEP 1647 H=1647:00 +CPU= 27.271 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:34:56 STEP 1648 H=1648:00 +CPU= 17.369 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:35:15 STEP 1649 H=1649:00 +CPU= 16.888 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:35:47 STEP 1650 H=1650:00 +CPU= 27.826 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:36:06 STEP 1651 H=1651:00 +CPU= 16.872 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:36:25 STEP 1652 H=1652:00 +CPU= 17.246 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:36:49 STEP 1653 H=1653:00 +CPU= 21.592 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:37:06 STEP 1654 H=1654:00 +CPU= 14.916 21:37:23 STEP 1655 H=1655:00 +CPU= 16.899 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:37:57 STEP 1656 H=1656:00 +CPU= 30.589 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001656 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001656 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001632 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001632 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001632 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001644 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001644 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001644 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_68.zip Uploading the intermediate file: upload_file_68.zip 21:38:15 STEP 1657 H=1657:00 +CPU= 14.491 21:38:30 STEP 1658 H=1658:00 +CPU= 15.355 21:38:56 STEP 1659 H=1659:00 +CPU= 26.409 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:39:15 STEP 1660 H=1660:00 +CPU= 16.888 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:39:34 STEP 1661 H=1661:00 +CPU= 16.827 21:40:02 STEP 1662 H=1662:00 +CPU= 27.725 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:40:20 STEP 1663 H=1663:00 +CPU= 16.236 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:40:39 STEP 1664 H=1664:00 +CPU= 17.251 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:41:10 STEP 1665 H=1665:00 +CPU= 27.194 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:41:30 STEP 1666 H=1666:00 +CPU= 17.328 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:41:49 STEP 1667 H=1667:00 +CPU= 16.853 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:42:21 STEP 1668 H=1668:00 +CPU= 30.334 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001668 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001668 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001668 21:42:40 STEP 1669 H=1669:00 +CPU= 16.847 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:42:59 STEP 1670 H=1670:00 +CPU= 17.320 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:43:31 STEP 1671 H=1671:00 +CPU= 27.159 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:43:50 STEP 1672 H=1672:00 +CPU= 17.149 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:44:08 STEP 1673 H=1673:00 +CPU= 16.416 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:44:37 STEP 1674 H=1674:00 +CPU= 26.998 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:44:56 STEP 1675 H=1675:00 +CPU= 16.387 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:45:14 STEP 1676 H=1676:00 +CPU= 16.864 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:45:45 STEP 1677 H=1677:00 +CPU= 26.465 21:46:02 STEP 1678 H=1678:00 +CPU= 16.836 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:46:21 STEP 1679 H=1679:00 +CPU= 17.228 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:46:57 STEP 1680 H=1680:00 +CPU= 31.644 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001680 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001680 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001656 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001668 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001668 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001668 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_69.zip Uploading the intermediate file: upload_file_69.zip 21:47:16 STEP 1681 H=1681:00 +CPU= 16.914 21:47:33 STEP 1682 H=1682:00 +CPU= 17.363 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:48:02 STEP 1683 H=1683:00 +CPU= 27.337 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:48:22 STEP 1684 H=1684:00 +CPU= 17.408 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:48:40 STEP 1685 H=1685:00 +CPU= 16.515 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:49:12 STEP 1686 H=1686:00 +CPU= 27.864 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:49:31 STEP 1687 H=1687:00 +CPU= 16.954 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:49:51 STEP 1688 H=1688:00 +CPU= 17.400 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:50:20 STEP 1689 H=1689:00 +CPU= 27.355 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:50:39 STEP 1690 H=1690:00 +CPU= 17.383 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:50:58 STEP 1691 H=1691:00 +CPU= 16.840 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:51:33 STEP 1692 H=1692:00 +CPU= 30.460 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001692 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001692 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001692 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:51:52 STEP 1693 H=1693:00 +CPU= 16.944 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:52:11 STEP 1694 H=1694:00 +CPU= 17.439 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:52:41 STEP 1695 H=1695:00 +CPU= 27.346 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:53:00 STEP 1696 H=1696:00 +CPU= 17.394 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:53:19 STEP 1697 H=1697:00 +CPU= 16.946 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:53:51 STEP 1698 H=1698:00 +CPU= 27.767 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:54:10 STEP 1699 H=1699:00 +CPU= 16.924 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:54:29 STEP 1700 H=1700:00 +CPU= 17.402 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:54:59 STEP 1701 H=1701:00 +CPU= 27.344 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:55:18 STEP 1702 H=1702:00 +CPU= 17.358 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:55:37 STEP 1703 H=1703:00 +CPU= 17.354 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:56:13 STEP 1704 H=1704:00 +CPU= 31.471 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001704 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001704 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001680 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001692 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001692 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001692 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_70.zip Uploading the intermediate file: upload_file_70.zip 21:56:30 STEP 1705 H=1705:00 +CPU= 16.775 Uploading trickle at timestep: 6134400 21:56:47 STEP 1706 H=1706:00 +CPU= 17.182 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:57:16 STEP 1707 H=1707:00 +CPU= 27.126 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:57:35 STEP 1708 H=1708:00 +CPU= 17.284 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:57:54 STEP 1709 H=1709:00 +CPU= 16.843 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:58:26 STEP 1710 H=1710:00 +CPU= 27.719 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:58:45 STEP 1711 H=1711:00 +CPU= 16.840 21:59:02 STEP 1712 H=1712:00 +CPU= 17.299 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:59:33 STEP 1713 H=1713:00 +CPU= 27.157 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:59:53 STEP 1714 H=1714:00 +CPU= 17.296 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1995050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0435 (argv4) batchid: 964 (argv5) wuid: 12181079 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /mnt/nvme/scratch1/boinc/slots/1 Project directory is: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079 ..mkdir for temp folder for results failed Copying: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_e2bfb140cd8aa1e91ea7e967da64f967 to: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_0435_1995050100_123_964_12181079.zip Unzipping the namelist zip file: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_ps_0435_1995050100_123_964_12181079.zip ic_ancil_file: ic_ancil_12181079 ifsdata_file: ifsdata_12181079 climate_data_file: clim_data_12181079 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_6daf3f584971cf80a09bf127d96865c7 to: /mnt/nvme/scratch1/boinc/slots/1/ic_ancil_12181079.zip Unzipping the IC ancils zip file: /mnt/nvme/scratch1/boinc/slots/1/ic_ancil_12181079.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_577a197af9d8e1bfbd56e77858179c57 to: /mnt/nvme/scratch1/boinc/slots/1/ifsdata/ifsdata_12181079.zip Unzipping the ifsdata_zip file: /mnt/nvme/scratch1/boinc/slots/1/ifsdata/ifsdata_12181079.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/nvme/scratch1/boinc/slots/1/159l_2/clim_data_12181079.zip Unzipping the climate data zip file: /mnt/nvme/scratch1/boinc/slots/1/159l_2/clim_data_12181079.zip Checking for progress XML file: /mnt/nvme/scratch1/boinc/slots/1/progress_file_12181079.xml Opened progress file ok : /mnt/nvme/scratch1/boinc/slots/1/progress_file_12181079.xml -- Model is restarting -- Adjusting last_iter, 1715, to previous model restart step. Creating progress file: /mnt/nvme/scratch1/boinc/slots/1/progress_file_12181079.xml last_cpu_time: 37910 upload_file_number: 71 last_iter: 1703 last_upload: 6134400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097 The child process has been launched with process id: 23373 Executing the command: /mnt/nvme/scratch1/boinc/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ormal.auroraborealis:1:1:23373:23373] [20221230:080004:1672383604:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:00:35 STEP 1704 H=1704:00 +CPU= 27.447 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:00:51 STEP 1705 H=1705:00 +CPU= 13.798 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:01:09 STEP 1706 H=1706:00 +CPU= 16.356 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:01:39 STEP 1707 H=1707:00 +CPU= 27.664 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:01:59 STEP 1708 H=1708:00 +CPU= 17.846 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:02:18 STEP 1709 H=1709:00 +CPU= 17.416 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:02:50 STEP 1710 H=1710:00 +CPU= 28.019 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:03:09 STEP 1711 H=1711:00 +CPU= 16.956 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:03:29 STEP 1712 H=1712:00 +CPU= 17.413 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:03:55 STEP 1713 H=1713:00 +CPU= 24.615 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:04:12 STEP 1714 H=1714:00 +CPU= 14.171 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:04:27 STEP 1715 H=1715:00 +CPU= 13.839 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:04:59 STEP 1716 H=1716:00 +CPU= 29.192 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001716 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001716 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001716 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:05:18 STEP 1717 H=1717:00 +CPU= 16.902 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:05:35 STEP 1718 H=1718:00 +CPU= 14.956 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:06:02 STEP 1719 H=1719:00 +CPU= 25.427 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:06:21 STEP 1720 H=1720:00 +CPU= 16.619 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:06:39 STEP 1721 H=1721:00 +CPU= 16.775 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:07:11 STEP 1722 H=1722:00 +CPU= 27.688 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:07:30 STEP 1723 H=1723:00 +CPU= 16.880 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:07:49 STEP 1724 H=1724:00 +CPU= 17.392 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:08:19 STEP 1725 H=1725:00 +CPU= 27.228 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:08:39 STEP 1726 H=1726:00 +CPU= 17.776 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:08:58 STEP 1727 H=1727:00 +CPU= 17.752 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:09:35 STEP 1728 H=1728:00 +CPU= 32.410 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001728 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001728 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001704 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001716 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001716 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001716 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_71.zip Uploading the intermediate file: upload_file_71.zip 08:09:52 STEP 1729 H=1729:00 +CPU= 17.171 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:10:12 STEP 1730 H=1730:00 +CPU= 17.837 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:10:42 STEP 1731 H=1731:00 +CPU= 27.848 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:11:01 STEP 1732 H=1732:00 +CPU= 17.381 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:11:20 STEP 1733 H=1733:00 +CPU= 17.427 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:11:53 STEP 1734 H=1734:00 +CPU= 28.456 08:12:10 STEP 1735 H=1735:00 +CPU= 17.239 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:12:30 STEP 1736 H=1736:00 +CPU= 17.919 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:13:00 STEP 1737 H=1737:00 +CPU= 27.851 08:13:18 STEP 1738 H=1738:00 +CPU= 17.834 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:13:37 STEP 1739 H=1739:00 +CPU= 17.466 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:14:12 STEP 1740 H=1740:00 +CPU= 30.923 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001740 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001740 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001740 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:14:32 STEP 1741 H=1741:00 +CPU= 17.436 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:14:52 STEP 1742 H=1742:00 +CPU= 17.843 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:15:22 STEP 1743 H=1743:00 +CPU= 27.869 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:15:41 STEP 1744 H=1744:00 +CPU= 17.787 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:16:01 STEP 1745 H=1745:00 +CPU= 17.230 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:16:33 STEP 1746 H=1746:00 +CPU= 28.218 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:16:52 STEP 1747 H=1747:00 +CPU= 17.110 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:17:12 STEP 1748 H=1748:00 +CPU= 17.531 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:17:41 STEP 1749 H=1749:00 +CPU= 27.397 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:18:01 STEP 1750 H=1750:00 +CPU= 17.831 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:18:21 STEP 1751 H=1751:00 +CPU= 17.457 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:18:57 STEP 1752 H=1752:00 +CPU= 32.091 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001752 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001752 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001728 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001740 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001740 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001740 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_72.zip Uploading the intermediate file: upload_file_72.zip 08:19:14 STEP 1753 H=1753:00 +CPU= 17.352 08:19:32 STEP 1754 H=1754:00 +CPU= 17.792 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:20:02 STEP 1755 H=1755:00 +CPU= 27.884 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:20:22 STEP 1756 H=1756:00 +CPU= 17.719 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:20:41 STEP 1757 H=1757:00 +CPU= 17.267 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:21:13 STEP 1758 H=1758:00 +CPU= 27.875 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:21:32 STEP 1759 H=1759:00 +CPU= 17.512 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:21:52 STEP 1760 H=1760:00 +CPU= 17.790 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:22:24 STEP 1761 H=1761:00 +CPU= 28.116 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:22:45 STEP 1762 H=1762:00 +CPU= 18.142 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:23:04 STEP 1763 H=1763:00 +CPU= 17.667 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:23:38 STEP 1764 H=1764:00 +CPU= 31.173 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001764 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001764 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001764 08:23:57 STEP 1765 H=1765:00 +CPU= 17.294 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:24:17 STEP 1766 H=1766:00 +CPU= 17.967 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:24:49 STEP 1767 H=1767:00 +CPU= 28.066 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:25:08 STEP 1768 H=1768:00 +CPU= 16.903 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:25:27 STEP 1769 H=1769:00 +CPU= 17.111 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:25:58 STEP 1770 H=1770:00 +CPU= 28.632 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:26:17 STEP 1771 H=1771:00 +CPU= 17.206 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:26:37 STEP 1772 H=1772:00 +CPU= 17.836 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:27:08 STEP 1773 H=1773:00 +CPU= 26.915 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:27:28 STEP 1774 H=1774:00 +CPU= 17.845 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:27:47 STEP 1775 H=1775:00 +CPU= 17.714 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:28:21 STEP 1776 H=1776:00 +CPU= 31.650 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001776 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001776 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001752 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001764 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001764 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001764 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_73.zip Uploading the intermediate file: upload_file_73.zip 08:28:40 STEP 1777 H=1777:00 +CPU= 17.341 08:28:58 STEP 1778 H=1778:00 +CPU= 17.891 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:29:30 STEP 1779 H=1779:00 +CPU= 27.928 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:29:50 STEP 1780 H=1780:00 +CPU= 17.857 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:30:10 STEP 1781 H=1781:00 +CPU= 17.223 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:30:40 STEP 1782 H=1782:00 +CPU= 28.541 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:31:00 STEP 1783 H=1783:00 +CPU= 17.563 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:31:20 STEP 1784 H=1784:00 +CPU= 18.148 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:31:52 STEP 1785 H=1785:00 +CPU= 27.911 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:32:12 STEP 1786 H=1786:00 +CPU= 17.676 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:32:32 STEP 1787 H=1787:00 +CPU= 17.465 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:33:07 STEP 1788 H=1788:00 +CPU= 31.230 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001788 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001788 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001788 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:33:26 STEP 1789 H=1789:00 +CPU= 17.441 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:33:46 STEP 1790 H=1790:00 +CPU= 17.860 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:34:16 STEP 1791 H=1791:00 +CPU= 28.216 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:34:37 STEP 1792 H=1792:00 +CPU= 18.025 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:34:56 STEP 1793 H=1793:00 +CPU= 17.490 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:35:29 STEP 1794 H=1794:00 +CPU= 28.631 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:35:48 STEP 1795 H=1795:00 +CPU= 17.444 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:36:08 STEP 1796 H=1796:00 +CPU= 17.880 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:36:38 STEP 1797 H=1797:00 +CPU= 27.549 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:36:57 STEP 1798 H=1798:00 +CPU= 17.518 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:37:17 STEP 1799 H=1799:00 +CPU= 17.534 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:37:54 STEP 1800 H=1800:00 +CPU= 32.744 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001800 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001800 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001776 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001788 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001788 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001788 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_74.zip Uploading the intermediate file: upload_file_74.zip 08:38:11 STEP 1801 H=1801:00 +CPU= 17.477 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:38:31 STEP 1802 H=1802:00 +CPU= 17.953 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:39:03 STEP 1803 H=1803:00 +CPU= 28.182 08:39:21 STEP 1804 H=1804:00 +CPU= 17.486 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:39:40 STEP 1805 H=1805:00 +CPU= 17.220 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:40:12 STEP 1806 H=1806:00 +CPU= 27.931 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:40:31 STEP 1807 H=1807:00 +CPU= 17.221 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:40:51 STEP 1808 H=1808:00 +CPU= 17.651 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:41:18 STEP 1809 H=1809:00 +CPU= 25.398 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:41:37 STEP 1810 H=1810:00 +CPU= 16.699 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:41:56 STEP 1811 H=1811:00 +CPU= 17.219 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:42:31 STEP 1812 H=1812:00 +CPU= 30.749 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001812 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001812 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001812 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:42:51 STEP 1813 H=1813:00 +CPU= 17.560 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:43:11 STEP 1814 H=1814:00 +CPU= 18.091 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:43:41 STEP 1815 H=1815:00 +CPU= 28.172 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:44:01 STEP 1816 H=1816:00 +CPU= 18.033 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:44:21 STEP 1817 H=1817:00 +CPU= 17.534 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:44:53 STEP 1818 H=1818:00 +CPU= 28.500 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:45:13 STEP 1819 H=1819:00 +CPU= 17.460 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:45:33 STEP 1820 H=1820:00 +CPU= 17.966 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:46:05 STEP 1821 H=1821:00 +CPU= 28.021 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:46:25 STEP 1822 H=1822:00 +CPU= 17.881 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:46:45 STEP 1823 H=1823:00 +CPU= 17.917 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:47:20 STEP 1824 H=1824:00 +CPU= 32.887 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001824 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001824 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001800 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001812 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001812 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001812 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_75.zip Uploading the intermediate file: upload_file_75.zip 08:47:37 STEP 1825 H=1825:00 +CPU= 17.491 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:47:57 STEP 1826 H=1826:00 +CPU= 18.112 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:48:29 STEP 1827 H=1827:00 +CPU= 28.082 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:48:50 STEP 1828 H=1828:00 +CPU= 18.099 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:49:09 STEP 1829 H=1829:00 +CPU= 17.592 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:49:40 STEP 1830 H=1830:00 +CPU= 28.686 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:49:59 STEP 1831 H=1831:00 +CPU= 17.445 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:50:20 STEP 1832 H=1832:00 +CPU= 17.945 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:50:52 STEP 1833 H=1833:00 +CPU= 28.048 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:51:12 STEP 1834 H=1834:00 +CPU= 17.994 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:51:32 STEP 1835 H=1835:00 +CPU= 17.439 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:52:07 STEP 1836 H=1836:00 +CPU= 31.237 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001836 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001836 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001836 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:52:27 STEP 1837 H=1837:00 +CPU= 17.536 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:52:46 STEP 1838 H=1838:00 +CPU= 17.849 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:53:17 STEP 1839 H=1839:00 +CPU= 28.056 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:53:36 STEP 1840 H=1840:00 +CPU= 17.958 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:53:56 STEP 1841 H=1841:00 +CPU= 17.527 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:54:28 STEP 1842 H=1842:00 +CPU= 27.864 08:54:45 STEP 1843 H=1843:00 +CPU= 16.925 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:55:05 STEP 1844 H=1844:00 +CPU= 17.619 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:55:35 STEP 1845 H=1845:00 +CPU= 28.033 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:55:55 STEP 1846 H=1846:00 +CPU= 17.839 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:56:15 STEP 1847 H=1847:00 +CPU= 18.026 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:56:51 STEP 1848 H=1848:00 +CPU= 32.734 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001848 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001848 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001824 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001836 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001836 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001836 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_76.zip Uploading the intermediate file: upload_file_76.zip 08:57:09 STEP 1849 H=1849:00 +CPU= 17.358 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:57:29 STEP 1850 H=1850:00 +CPU= 17.938 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:57:59 STEP 1851 H=1851:00 +CPU= 28.042 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:58:19 STEP 1852 H=1852:00 +CPU= 17.875 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:58:39 STEP 1853 H=1853:00 +CPU= 17.584 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:59:11 STEP 1854 H=1854:00 +CPU= 28.680 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:59:31 STEP 1855 H=1855:00 +CPU= 17.715 Suspend request received from the BOINC client, suspending the child process Resuming the child process 08:59:52 STEP 1856 H=1856:00 +CPU= 18.252 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:00:24 STEP 1857 H=1857:00 +CPU= 28.294 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:00:44 STEP 1858 H=1858:00 +CPU= 18.105 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:01:04 STEP 1859 H=1859:00 +CPU= 17.751 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:01:37 STEP 1860 H=1860:00 +CPU= 30.997 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001860 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001860 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001860 09:01:56 STEP 1861 H=1861:00 +CPU= 17.568 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:02:17 STEP 1862 H=1862:00 +CPU= 18.146 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:02:49 STEP 1863 H=1863:00 +CPU= 28.241 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:03:09 STEP 1864 H=1864:00 +CPU= 18.028 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:03:29 STEP 1865 H=1865:00 +CPU= 17.686 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:03:59 STEP 1866 H=1866:00 +CPU= 28.784 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:04:18 STEP 1867 H=1867:00 +CPU= 16.655 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:04:38 STEP 1868 H=1868:00 +CPU= 18.140 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:05:11 STEP 1869 H=1869:00 +CPU= 28.205 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:05:31 STEP 1870 H=1870:00 +CPU= 18.123 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:05:51 STEP 1871 H=1871:00 +CPU= 18.074 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:06:28 STEP 1872 H=1872:00 +CPU= 33.041 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMGGhq0f+001872 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMSHhq0f+001872 Moving to projects directory: /mnt/nvme/scratch1/boinc/slots/1/ICMUAhq0f+001872 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001848 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMGGhq0f+001860 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMSHhq0f+001860 Adding to the zip: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_12181079/ICMUAhq0f+001860 Zipping up the intermediate file: /mnt/nvme/scratch1/boinc/projects/climateprediction.net/oifs_43r3_ps_0435_1995050100_123_964_12181079_0_r71764097_77.zip Uploading the intermediate file: upload_file_77.zip 09:06:45 STEP 1873 H=1873:00 +CPU= 17.421 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:07:06 STEP 1874 H=1874:00 +CPU= 18.108 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:07:36 STEP 1875 H=1875:00 +CPU= 28.109 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:07:55 STEP 1876 H=1876:00 +CPU= 17.632 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:08:15 STEP 1877 H=1877:00 +CPU= 17.843 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:08:48 STEP 1878 H=1878:00 +CPU= 28.866 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:09:08 STEP 1879 H=1879:00 +CPU= 17.689 Suspend request received from the BOINC client, suspending the child process Resuming the child process 09:09:29 STEP 1880 H=1880:00 +CPU= 18.002 Suspend request received from the BOINC client, suspending the child process Resuming the child process Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:02:04 | 1489681 | 22266093 | oifs_43r3_ps_0435_1995050100_123_964_12181079_0 | 6,134,400 | 37,710 | 0.0061 |
09 May 2023 16:01:59 | 1489681 | 22266093 | oifs_43r3_ps_0435_1995050100_123_964_12181079_0 | 5,270,400 | 32,729 | 0.0062 |
09 May 2023 16:01:59 | 1489681 | 22266093 | oifs_43r3_ps_0435_1995050100_123_964_12181079_0 | 5,270,400 | 32,729 | 0.0062 |
09 May 2023 15:57:06 | 1489681 | 22266093 | oifs_43r3_ps_0435_1995050100_123_964_12181079_0 | 1,728,000 | 10,583 | 0.0061 |
09 May 2023 15:57:01 | 1489681 | 22266093 | oifs_43r3_ps_0435_1995050100_123_964_12181079_0 | 864,000 | 5,361 | 0.0062 |
©2024 cpdn.org