Name | oifs_43r3_ps_0210_1997050100_123_966_12182854_0 |
Workunit | 12182854 |
Created | 22 Dec 2022, 13:00:10 UTC |
Sent | 22 Dec 2022, 13:02:33 UTC |
Report deadline | 21 Jan 2023, 13:02:33 UTC |
Received | 23 Dec 2022, 7:54:31 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1509499 |
Run time | 7 hours 50 min 12 sec |
CPU time | 7 hours 36 min 31 sec |
Validate state | Invalid |
Credit | 5,201.28 |
Device peak FLOPS | 5.06 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,413.59 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 5,259.73 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> 000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000708 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_29.zip Uploading the intermediate file: upload_file_29.zip 19:32:31 STEP 721 H= 721:00 +CPU= 18.192 19:32:50 STEP 722 H= 722:00 +CPU= 18.333 19:33:19 STEP 723 H= 723:00 +CPU= 28.913 19:33:39 STEP 724 H= 724:00 +CPU= 19.032 19:33:57 STEP 725 H= 725:00 +CPU= 18.052 19:34:28 STEP 726 H= 726:00 +CPU= 29.779 19:34:47 STEP 727 H= 727:00 +CPU= 18.351 19:35:06 STEP 728 H= 728:00 +CPU= 18.779 19:35:35 STEP 729 H= 729:00 +CPU= 28.555 19:35:53 STEP 730 H= 730:00 +CPU= 18.451 19:36:11 STEP 731 H= 731:00 +CPU= 17.549 19:36:42 STEP 732 H= 732:00 +CPU= 31.167 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000732 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000732 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000732 19:37:00 STEP 733 H= 733:00 +CPU= 17.784 19:37:18 STEP 734 H= 734:00 +CPU= 18.059 19:37:47 STEP 735 H= 735:00 +CPU= 28.318 19:38:05 STEP 736 H= 736:00 +CPU= 18.263 19:38:23 STEP 737 H= 737:00 +CPU= 17.867 19:38:54 STEP 738 H= 738:00 +CPU= 29.594 19:39:13 STEP 739 H= 739:00 +CPU= 18.396 19:39:31 STEP 740 H= 740:00 +CPU= 18.244 19:39:59 STEP 741 H= 741:00 +CPU= 28.174 19:40:18 STEP 742 H= 742:00 +CPU= 18.221 19:40:36 STEP 743 H= 743:00 +CPU= 18.343 19:41:11 STEP 744 H= 744:00 +CPU= 33.632 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000744 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000744 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000732 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_30.zip Uploading the intermediate file: upload_file_30.zip 19:41:28 STEP 745 H= 745:00 +CPU= 17.759 19:41:47 STEP 746 H= 746:00 +CPU= 18.186 19:42:15 STEP 747 H= 747:00 +CPU= 28.152 19:42:33 STEP 748 H= 748:00 +CPU= 18.265 19:42:51 STEP 749 H= 749:00 +CPU= 17.545 19:43:20 STEP 750 H= 750:00 +CPU= 28.786 19:43:38 STEP 751 H= 751:00 +CPU= 17.834 19:43:56 STEP 752 H= 752:00 +CPU= 18.083 19:44:24 STEP 753 H= 753:00 +CPU= 28.208 19:44:42 STEP 754 H= 754:00 +CPU= 18.261 19:45:00 STEP 755 H= 755:00 +CPU= 17.549 19:45:31 STEP 756 H= 756:00 +CPU= 31.184 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000756 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000756 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000756 19:45:49 STEP 757 H= 757:00 +CPU= 17.780 19:46:07 STEP 758 H= 758:00 +CPU= 18.067 19:46:36 STEP 759 H= 759:00 +CPU= 28.344 19:46:54 STEP 760 H= 760:00 +CPU= 18.267 19:47:12 STEP 761 H= 761:00 +CPU= 17.499 19:47:41 STEP 762 H= 762:00 +CPU= 28.837 19:47:58 STEP 763 H= 763:00 +CPU= 17.674 19:48:17 STEP 764 H= 764:00 +CPU= 18.315 19:48:45 STEP 765 H= 765:00 +CPU= 28.204 19:49:04 STEP 766 H= 766:00 +CPU= 18.184 19:49:22 STEP 767 H= 767:00 +CPU= 18.018 19:49:55 STEP 768 H= 768:00 +CPU= 33.251 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000768 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000768 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000756 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_31.zip Uploading the intermediate file: upload_file_31.zip 19:50:13 STEP 769 H= 769:00 +CPU= 17.767 19:50:31 STEP 770 H= 770:00 +CPU= 18.066 19:50:59 STEP 771 H= 771:00 +CPU= 28.261 19:51:18 STEP 772 H= 772:00 +CPU= 18.250 19:51:35 STEP 773 H= 773:00 +CPU= 17.538 19:52:04 STEP 774 H= 774:00 +CPU= 28.838 19:52:22 STEP 775 H= 775:00 +CPU= 17.784 19:52:40 STEP 776 H= 776:00 +CPU= 18.108 19:53:09 STEP 777 H= 777:00 +CPU= 28.206 19:53:27 STEP 778 H= 778:00 +CPU= 18.239 19:53:45 STEP 779 H= 779:00 +CPU= 17.541 19:54:16 STEP 780 H= 780:00 +CPU= 31.161 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000780 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000780 19:54:34 STEP 781 H= 781:00 +CPU= 17.747 19:54:52 STEP 782 H= 782:00 +CPU= 18.060 19:55:20 STEP 783 H= 783:00 +CPU= 28.390 19:55:39 STEP 784 H= 784:00 +CPU= 18.241 19:55:56 STEP 785 H= 785:00 +CPU= 17.509 19:56:25 STEP 786 H= 786:00 +CPU= 28.717 19:56:43 STEP 787 H= 787:00 +CPU= 17.660 19:57:01 STEP 788 H= 788:00 +CPU= 18.065 19:57:29 STEP 789 H= 789:00 +CPU= 28.147 19:57:47 STEP 790 H= 790:00 +CPU= 18.178 19:58:05 STEP 791 H= 791:00 +CPU= 17.994 19:58:38 STEP 792 H= 792:00 +CPU= 32.972 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000792 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000792 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000792 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000780 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000780 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000780 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_32.zip Uploading the intermediate file: upload_file_32.zip 19:58:56 STEP 793 H= 793:00 +CPU= 17.744 19:59:14 STEP 794 H= 794:00 +CPU= 18.139 19:59:43 STEP 795 H= 795:00 +CPU= 28.190 20:00:01 STEP 796 H= 796:00 +CPU= 18.236 20:00:19 STEP 797 H= 797:00 +CPU= 17.557 20:00:48 STEP 798 H= 798:00 +CPU= 28.830 20:01:05 STEP 799 H= 799:00 +CPU= 17.761 20:01:24 STEP 800 H= 800:00 +CPU= 18.104 20:01:52 STEP 801 H= 801:00 +CPU= 28.194 20:02:10 STEP 802 H= 802:00 +CPU= 18.395 20:02:28 STEP 803 H= 803:00 +CPU= 17.540 20:02:59 STEP 804 H= 804:00 +CPU= 31.129 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000804 20:03:17 STEP 805 H= 805:00 +CPU= 17.758 20:03:35 STEP 806 H= 806:00 +CPU= 18.036 20:04:03 STEP 807 H= 807:00 +CPU= 28.210 20:04:22 STEP 808 H= 808:00 +CPU= 18.234 20:04:39 STEP 809 H= 809:00 +CPU= 17.544 20:05:08 STEP 810 H= 810:00 +CPU= 28.742 20:05:26 STEP 811 H= 811:00 +CPU= 17.668 20:05:44 STEP 812 H= 812:00 +CPU= 18.128 20:06:12 STEP 813 H= 813:00 +CPU= 28.152 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1997050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0210 (argv4) batchid: 966 (argv5) wuid: 12182854 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_d8b00b9344959ed1ed7a16af55a49efd to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip ic_ancil_file: ic_ancil_12182854 ifsdata_file: ifsdata_12182854 climate_data_file: clim_data_12182854 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_93925be62ffebdba57ac843b302af8cb to: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_50410fe7985a2f73445c6a2671fc832d to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12182854.xml -- Model is restarting -- Adjusting last_iter, 813, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml last_cpu_time: 19271 upload_file_number: 33 last_iter: 791 last_upload: 2851200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993 The child process has been launched with process id: 23004 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 20:07:56 STEP 792 H= 792:00 +CPU= 31.556 20:08:14 STEP 793 H= 793:00 +CPU= 17.722 20:08:32 STEP 794 H= 794:00 +CPU= 18.269 20:09:00 STEP 795 H= 795:00 +CPU= 28.075 20:09:18 STEP 796 H= 796:00 +CPU= 18.269 20:09:37 STEP 797 H= 797:00 +CPU= 18.104 20:10:06 STEP 798 H= 798:00 +CPU= 28.998 20:10:24 STEP 799 H= 799:00 +CPU= 18.057 20:10:43 STEP 800 H= 800:00 +CPU= 18.504 20:11:11 STEP 801 H= 801:00 +CPU= 28.167 20:11:30 STEP 802 H= 802:00 +CPU= 18.310 20:11:48 STEP 803 H= 803:00 +CPU= 17.810 20:12:19 STEP 804 H= 804:00 +CPU= 31.383 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000804 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000804 20:12:37 STEP 805 H= 805:00 +CPU= 17.832 20:12:56 STEP 806 H= 806:00 +CPU= 18.391 20:13:24 STEP 807 H= 807:00 +CPU= 28.310 20:13:43 STEP 808 H= 808:00 +CPU= 18.506 20:14:01 STEP 809 H= 809:00 +CPU= 17.862 20:14:30 STEP 810 H= 810:00 +CPU= 28.941 20:14:48 STEP 811 H= 811:00 +CPU= 17.787 20:15:06 STEP 812 H= 812:00 +CPU= 18.297 20:15:34 STEP 813 H= 813:00 +CPU= 28.098 20:15:53 STEP 814 H= 814:00 +CPU= 18.277 20:16:11 STEP 815 H= 815:00 +CPU= 18.284 20:16:44 STEP 816 H= 816:00 +CPU= 32.926 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000816 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000816 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000792 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000792 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000792 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000804 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000804 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000804 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_33.zip Uploading the intermediate file: upload_file_33.zip 20:17:02 STEP 817 H= 817:00 +CPU= 17.695 20:17:20 STEP 818 H= 818:00 +CPU= 18.357 20:17:48 STEP 819 H= 819:00 +CPU= 28.027 20:18:07 STEP 820 H= 820:00 +CPU= 18.284 20:18:25 STEP 821 H= 821:00 +CPU= 17.924 20:18:54 STEP 822 H= 822:00 +CPU= 28.794 20:19:12 STEP 823 H= 823:00 +CPU= 17.957 20:19:31 STEP 824 H= 824:00 +CPU= 18.428 20:19:59 STEP 825 H= 825:00 +CPU= 28.346 20:20:17 STEP 826 H= 826:00 +CPU= 18.326 20:20:35 STEP 827 H= 827:00 +CPU= 17.895 20:21:07 STEP 828 H= 828:00 +CPU= 31.379 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000828 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000828 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000828 20:21:25 STEP 829 H= 829:00 +CPU= 17.937 20:21:44 STEP 830 H= 830:00 +CPU= 18.469 20:22:12 STEP 831 H= 831:00 +CPU= 28.283 20:22:31 STEP 832 H= 832:00 +CPU= 18.521 20:22:49 STEP 833 H= 833:00 +CPU= 17.968 20:23:18 STEP 834 H= 834:00 +CPU= 28.917 20:23:36 STEP 835 H= 835:00 +CPU= 17.820 20:23:55 STEP 836 H= 836:00 +CPU= 18.357 20:24:23 STEP 837 H= 837:00 +CPU= 28.306 20:24:41 STEP 838 H= 838:00 +CPU= 18.350 20:25:00 STEP 839 H= 839:00 +CPU= 18.328 20:25:33 STEP 840 H= 840:00 +CPU= 33.100 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000840 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000840 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000816 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000828 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000828 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000828 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_34.zip Uploading the intermediate file: upload_file_34.zip 20:25:51 STEP 841 H= 841:00 +CPU= 17.748 20:26:10 STEP 842 H= 842:00 +CPU= 18.469 20:26:38 STEP 843 H= 843:00 +CPU= 28.129 20:26:57 STEP 844 H= 844:00 +CPU= 18.501 20:27:15 STEP 845 H= 845:00 +CPU= 17.817 20:27:44 STEP 846 H= 846:00 +CPU= 29.020 20:28:02 STEP 847 H= 847:00 +CPU= 17.928 20:28:20 STEP 848 H= 848:00 +CPU= 18.411 20:28:49 STEP 849 H= 849:00 +CPU= 28.285 20:29:07 STEP 850 H= 850:00 +CPU= 18.466 20:29:25 STEP 851 H= 851:00 +CPU= 17.804 20:29:57 STEP 852 H= 852:00 +CPU= 31.130 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000852 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000852 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000852 20:30:15 STEP 853 H= 853:00 +CPU= 17.922 20:30:33 STEP 854 H= 854:00 +CPU= 18.428 20:31:01 STEP 855 H= 855:00 +CPU= 28.231 20:31:20 STEP 856 H= 856:00 +CPU= 18.366 20:31:38 STEP 857 H= 857:00 +CPU= 17.858 20:32:07 STEP 858 H= 858:00 +CPU= 28.855 20:32:25 STEP 859 H= 859:00 +CPU= 17.863 20:32:43 STEP 860 H= 860:00 +CPU= 18.301 20:33:12 STEP 861 H= 861:00 +CPU= 28.472 20:33:30 STEP 862 H= 862:00 +CPU= 18.235 20:33:49 STEP 863 H= 863:00 +CPU= 18.244 20:34:22 STEP 864 H= 864:00 +CPU= 32.925 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000864 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000864 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000840 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000852 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000852 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000852 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_35.zip Uploading the intermediate file: upload_file_35.zip 20:34:39 STEP 865 H= 865:00 +CPU= 17.698 20:34:58 STEP 866 H= 866:00 +CPU= 18.195 20:35:26 STEP 867 H= 867:00 +CPU= 28.134 20:35:44 STEP 868 H= 868:00 +CPU= 18.262 20:36:02 STEP 869 H= 869:00 +CPU= 17.736 20:36:31 STEP 870 H= 870:00 +CPU= 28.837 20:36:49 STEP 871 H= 871:00 +CPU= 17.742 20:37:07 STEP 872 H= 872:00 +CPU= 18.283 20:37:35 STEP 873 H= 873:00 +CPU= 28.213 20:37:54 STEP 874 H= 874:00 +CPU= 18.287 20:38:11 STEP 875 H= 875:00 +CPU= 17.801 20:38:43 STEP 876 H= 876:00 +CPU= 31.128 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000876 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000876 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000876 20:39:01 STEP 877 H= 877:00 +CPU= 18.060 20:39:20 STEP 878 H= 878:00 +CPU= 18.539 20:39:48 STEP 879 H= 879:00 +CPU= 28.159 20:40:06 STEP 880 H= 880:00 +CPU= 18.333 20:40:24 STEP 881 H= 881:00 +CPU= 17.778 20:40:53 STEP 882 H= 882:00 +CPU= 28.927 20:41:11 STEP 883 H= 883:00 +CPU= 17.739 20:41:30 STEP 884 H= 884:00 +CPU= 18.278 20:41:58 STEP 885 H= 885:00 +CPU= 28.192 20:42:16 STEP 886 H= 886:00 +CPU= 18.265 20:42:35 STEP 887 H= 887:00 +CPU= 18.289 20:43:08 STEP 888 H= 888:00 +CPU= 32.990 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000888 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000888 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000864 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000876 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000876 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_36.zip Uploading the intermediate file: upload_file_36.zip 20:43:26 STEP 889 H= 889:00 +CPU= 17.927 20:43:44 STEP 890 H= 890:00 +CPU= 18.321 20:44:12 STEP 891 H= 891:00 +CPU= 28.242 20:44:31 STEP 892 H= 892:00 +CPU= 18.322 20:44:49 STEP 893 H= 893:00 +CPU= 17.789 20:45:18 STEP 894 H= 894:00 +CPU= 29.019 20:45:36 STEP 895 H= 895:00 +CPU= 17.797 20:45:54 STEP 896 H= 896:00 +CPU= 18.347 20:46:22 STEP 897 H= 897:00 +CPU= 28.251 20:46:41 STEP 898 H= 898:00 +CPU= 18.346 20:46:59 STEP 899 H= 899:00 +CPU= 17.840 20:47:30 STEP 900 H= 900:00 +CPU= 31.142 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000900 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000900 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000900 20:47:48 STEP 901 H= 901:00 +CPU= 17.764 20:48:06 STEP 902 H= 902:00 +CPU= 18.384 20:48:35 STEP 903 H= 903:00 +CPU= 28.149 20:48:53 STEP 904 H= 904:00 +CPU= 18.364 20:49:11 STEP 905 H= 905:00 +CPU= 17.831 20:49:40 STEP 906 H= 906:00 +CPU= 28.858 20:49:58 STEP 907 H= 907:00 +CPU= 17.829 20:50:16 STEP 908 H= 908:00 +CPU= 18.299 20:50:44 STEP 909 H= 909:00 +CPU= 28.167 20:51:03 STEP 910 H= 910:00 +CPU= 18.815 20:51:22 STEP 911 H= 911:00 +CPU= 18.302 20:51:55 STEP 912 H= 912:00 +CPU= 33.136 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000912 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000912 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000888 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000900 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000900 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_37.zip Uploading the intermediate file: upload_file_37.zip 20:52:13 STEP 913 H= 913:00 +CPU= 17.822 20:52:31 STEP 914 H= 914:00 +CPU= 17.792 20:52:58 STEP 915 H= 915:00 +CPU= 26.901 20:53:15 STEP 916 H= 916:00 +CPU= 17.292 20:53:32 STEP 917 H= 917:00 +CPU= 17.243 20:54:01 STEP 918 H= 918:00 +CPU= 28.827 20:54:19 STEP 919 H= 919:00 +CPU= 17.800 20:54:38 STEP 920 H= 920:00 +CPU= 18.541 20:55:06 STEP 921 H= 921:00 +CPU= 28.287 20:55:25 STEP 922 H= 922:00 +CPU= 18.346 20:55:43 STEP 923 H= 923:00 +CPU= 17.828 20:56:14 STEP 924 H= 924:00 +CPU= 31.148 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000924 20:56:32 STEP 925 H= 925:00 +CPU= 17.777 20:56:50 STEP 926 H= 926:00 +CPU= 18.302 20:57:18 STEP 927 H= 927:00 +CPU= 28.179 20:57:37 STEP 928 H= 928:00 +CPU= 18.256 20:57:54 STEP 929 H= 929:00 +CPU= 17.742 20:58:23 STEP 930 H= 930:00 +CPU= 28.717 20:58:41 STEP 931 H= 931:00 +CPU= 17.783 20:58:59 STEP 932 H= 932:00 +CPU= 18.335 20:59:28 STEP 933 H= 933:00 +CPU= 28.159 20:59:46 STEP 934 H= 934:00 +CPU= 18.335 21:00:05 STEP 935 H= 935:00 +CPU= 18.333 21:00:38 STEP 936 H= 936:00 +CPU= 32.999 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000924 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_38.zip Uploading the intermediate file: upload_file_38.zip 21:00:56 STEP 937 H= 937:00 +CPU= 17.818 21:01:14 STEP 938 H= 938:00 +CPU= 18.354 21:01:42 STEP 939 H= 939:00 +CPU= 28.196 21:02:01 STEP 940 H= 940:00 +CPU= 18.372 21:02:19 STEP 941 H= 941:00 +CPU= 17.871 21:02:48 STEP 942 H= 942:00 +CPU= 28.814 21:03:06 STEP 943 H= 943:00 +CPU= 17.810 21:03:24 STEP 944 H= 944:00 +CPU= 18.341 21:03:52 STEP 945 H= 945:00 +CPU= 28.362 21:04:11 STEP 946 H= 946:00 +CPU= 18.369 21:04:29 STEP 947 H= 947:00 +CPU= 17.800 21:05:00 STEP 948 H= 948:00 +CPU= 31.193 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000948 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000948 21:05:18 STEP 949 H= 949:00 +CPU= 17.841 21:05:36 STEP 950 H= 950:00 +CPU= 18.354 21:06:05 STEP 951 H= 951:00 +CPU= 28.266 21:06:23 STEP 952 H= 952:00 +CPU= 18.365 21:06:41 STEP 953 H= 953:00 +CPU= 17.806 21:07:10 STEP 954 H= 954:00 +CPU= 28.767 21:07:28 STEP 955 H= 955:00 +CPU= 17.776 21:07:46 STEP 956 H= 956:00 +CPU= 18.267 21:08:14 STEP 957 H= 957:00 +CPU= 28.174 21:08:33 STEP 958 H= 958:00 +CPU= 18.300 21:08:51 STEP 959 H= 959:00 +CPU= 18.251 21:09:24 STEP 960 H= 960:00 +CPU= 32.855 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000960 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000960 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000948 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000948 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000948 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_39.zip Uploading the intermediate file: upload_file_39.zip 21:09:42 STEP 961 H= 961:00 +CPU= 17.758 21:10:00 STEP 962 H= 962:00 +CPU= 18.241 21:10:28 STEP 963 H= 963:00 +CPU= 28.110 21:10:47 STEP 964 H= 964:00 +CPU= 18.226 21:11:04 STEP 965 H= 965:00 +CPU= 17.732 21:11:33 STEP 966 H= 966:00 +CPU= 28.776 21:11:51 STEP 967 H= 967:00 +CPU= 17.802 21:12:10 STEP 968 H= 968:00 +CPU= 18.352 21:12:38 STEP 969 H= 969:00 +CPU= 28.175 21:12:56 STEP 970 H= 970:00 +CPU= 18.312 21:13:14 STEP 971 H= 971:00 +CPU= 17.758 21:13:45 STEP 972 H= 972:00 +CPU= 31.244 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000972 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000972 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000972 21:14:03 STEP 973 H= 973:00 +CPU= 17.784 21:14:22 STEP 974 H= 974:00 +CPU= 18.271 21:14:50 STEP 975 H= 975:00 +CPU= 28.316 21:15:09 STEP 976 H= 976:00 +CPU= 18.541 21:15:27 STEP 977 H= 977:00 +CPU= 17.753 21:15:56 STEP 978 H= 978:00 +CPU= 29.053 21:16:14 STEP 979 H= 979:00 +CPU= 17.931 21:16:32 STEP 980 H= 980:00 +CPU= 18.283 21:17:01 STEP 981 H= 981:00 +CPU= 28.352 21:17:19 STEP 982 H= 982:00 +CPU= 18.291 21:17:38 STEP 983 H= 983:00 +CPU= 18.398 21:18:11 STEP 984 H= 984:00 +CPU= 33.195 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000984 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000984 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000972 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_40.zip Uploading the intermediate file: upload_file_40.zip 21:18:29 STEP 985 H= 985:00 +CPU= 17.912 21:18:48 STEP 986 H= 986:00 +CPU= 18.220 21:19:16 STEP 987 H= 987:00 +CPU= 28.336 21:19:34 STEP 988 H= 988:00 +CPU= 18.295 21:19:52 STEP 989 H= 989:00 +CPU= 17.778 21:20:21 STEP 990 H= 990:00 +CPU= 28.838 21:20:39 STEP 991 H= 991:00 +CPU= 17.815 21:20:58 STEP 992 H= 992:00 +CPU= 18.406 21:21:26 STEP 993 H= 993:00 +CPU= 28.302 21:21:44 STEP 994 H= 994:00 +CPU= 18.362 21:22:02 STEP 995 H= 995:00 +CPU= 17.952 21:22:34 STEP 996 H= 996:00 +CPU= 31.501 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000996 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000996 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000996 21:22:52 STEP 997 H= 997:00 +CPU= 17.946 21:23:11 STEP 998 H= 998:00 +CPU= 18.376 21:23:39 STEP 999 H= 999:00 +CPU= 28.309 21:23:58 STEP 1000 H=1000:00 +CPU= 18.429 21:24:15 STEP 1001 H=1001:00 +CPU= 17.772 21:24:44 STEP 1002 H=1002:00 +CPU= 29.030 21:25:03 STEP 1003 H=1003:00 +CPU= 18.027 21:25:21 STEP 1004 H=1004:00 +CPU= 18.424 21:25:50 STEP 1005 H=1005:00 +CPU= 28.251 21:26:08 STEP 1006 H=1006:00 +CPU= 18.298 21:26:26 STEP 1007 H=1007:00 +CPU= 18.279 21:26:59 STEP 1008 H=1008:00 +CPU= 33.041 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001008 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001008 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000996 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_41.zip Uploading the intermediate file: upload_file_41.zip 21:27:17 STEP 1009 H=1009:00 +CPU= 17.826 21:27:36 STEP 1010 H=1010:00 +CPU= 18.262 21:28:04 STEP 1011 H=1011:00 +CPU= 28.422 21:28:22 STEP 1012 H=1012:00 +CPU= 18.383 21:28:40 STEP 1013 H=1013:00 +CPU= 17.715 21:29:09 STEP 1014 H=1014:00 +CPU= 28.912 21:29:27 STEP 1015 H=1015:00 +CPU= 17.838 21:29:46 STEP 1016 H=1016:00 +CPU= 18.330 21:30:14 STEP 1017 H=1017:00 +CPU= 28.293 21:30:32 STEP 1018 H=1018:00 +CPU= 18.369 21:30:50 STEP 1019 H=1019:00 +CPU= 17.814 21:31:21 STEP 1020 H=1020:00 +CPU= 31.204 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001020 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001020 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001020 21:31:39 STEP 1021 H=1021:00 +CPU= 17.863 21:31:58 STEP 1022 H=1022:00 +CPU= 18.307 21:32:26 STEP 1023 H=1023:00 +CPU= 28.316 21:32:45 STEP 1024 H=1024:00 +CPU= 18.360 21:33:02 STEP 1025 H=1025:00 +CPU= 17.810 21:33:31 STEP 1026 H=1026:00 +CPU= 28.812 21:33:49 STEP 1027 H=1027:00 +CPU= 17.744 21:34:07 STEP 1028 H=1028:00 +CPU= 18.263 21:34:36 STEP 1029 H=1029:00 +CPU= 28.371 21:34:56 STEP 1030 H=1030:00 +CPU= 19.427 21:35:15 STEP 1031 H=1031:00 +CPU= 18.697 21:35:49 STEP 1032 H=1032:00 +CPU= 33.170 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001032 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001032 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001020 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_42.zip Uploading the intermediate file: upload_file_42.zip 21:36:07 STEP 1033 H=1033:00 +CPU= 17.897 Uploading trickle at timestep: 3715200 21:36:26 STEP 1034 H=1034:00 +CPU= 18.344 21:36:54 STEP 1035 H=1035:00 +CPU= 28.476 21:37:14 STEP 1036 H=1036:00 +CPU= 18.957 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:37:43 STEP 1037 H=1037:00 +CPU= 18.341 21:38:13 STEP 1038 H=1038:00 +CPU= 29.159 21:38:31 STEP 1039 H=1039:00 +CPU= 17.874 21:38:49 STEP 1040 H=1040:00 +CPU= 18.186 21:39:18 STEP 1041 H=1041:00 +CPU= 28.497 21:39:36 STEP 1042 H=1042:00 +CPU= 18.384 21:39:54 STEP 1043 H=1043:00 +CPU= 17.696 21:40:26 STEP 1044 H=1044:00 +CPU= 31.432 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001044 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001044 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001044 21:40:44 STEP 1045 H=1045:00 +CPU= 18.034 21:41:08 STEP 1046 H=1046:00 +CPU= 19.782 21:41:40 STEP 1047 H=1047:00 +CPU= 31.060 Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:49:00 STEP 1048 H=1048:00 +CPU= 23.654 21:49:20 STEP 1049 H=1049:00 +CPU= 19.795 21:49:52 STEP 1050 H=1050:00 +CPU= 31.463 21:50:12 STEP 1051 H=1051:00 +CPU= 19.810 21:50:35 STEP 1052 H=1052:00 +CPU= 20.960 21:51:06 STEP 1053 H=1053:00 +CPU= 30.832 21:51:27 STEP 1054 H=1054:00 +CPU= 20.217 21:51:48 STEP 1055 H=1055:00 +CPU= 20.839 21:52:33 STEP 1056 H=1056:00 +CPU= 37.169 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001056 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001056 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001044 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001044 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001044 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_43.zip Uploading the intermediate file: upload_file_43.zip 21:52:54 STEP 1057 H=1057:00 +CPU= 19.936 21:53:16 STEP 1058 H=1058:00 +CPU= 20.469 21:53:49 STEP 1059 H=1059:00 +CPU= 31.600 21:54:10 STEP 1060 H=1060:00 +CPU= 20.357 21:54:30 STEP 1061 H=1061:00 +CPU= 19.700 21:55:03 STEP 1062 H=1062:00 +CPU= 31.923 21:55:26 STEP 1063 H=1063:00 +CPU= 20.921 21:55:47 STEP 1064 H=1064:00 +CPU= 20.484 21:56:19 STEP 1065 H=1065:00 +CPU= 31.098 21:56:40 STEP 1066 H=1066:00 +CPU= 20.535 21:57:00 STEP 1067 H=1067:00 +CPU= 19.741 21:57:36 STEP 1068 H=1068:00 +CPU= 34.644 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068 21:57:57 STEP 1069 H=1069:00 +CPU= 19.966 21:58:17 STEP 1070 H=1070:00 +CPU= 19.662 21:58:48 STEP 1071 H=1071:00 +CPU= 30.841 21:59:08 STEP 1072 H=1072:00 +CPU= 20.348 21:59:28 STEP 1073 H=1073:00 +CPU= 19.785 22:00:01 STEP 1074 H=1074:00 +CPU= 31.445 22:00:21 STEP 1075 H=1075:00 +CPU= 19.734 22:00:42 STEP 1076 H=1076:00 +CPU= 20.439 22:01:14 STEP 1077 H=1077:00 +CPU= 30.924 22:01:35 STEP 1078 H=1078:00 +CPU= 20.489 22:01:57 STEP 1079 H=1079:00 +CPU= 20.709 22:02:35 STEP 1080 H=1080:00 +CPU= 36.156 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001080 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001080 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001068 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_44.zip Uploading the intermediate file: upload_file_44.zip 22:02:56 STEP 1081 H=1081:00 +CPU= 19.818 22:03:17 STEP 1082 H=1082:00 +CPU= 20.453 22:03:48 STEP 1083 H=1083:00 +CPU= 31.014 22:04:09 STEP 1084 H=1084:00 +CPU= 20.255 22:04:30 STEP 1085 H=1085:00 +CPU= 19.924 22:05:02 STEP 1086 H=1086:00 +CPU= 31.577 22:05:22 STEP 1087 H=1087:00 +CPU= 19.775 22:05:43 STEP 1088 H=1088:00 +CPU= 20.551 22:06:14 STEP 1089 H=1089:00 +CPU= 30.603 22:06:35 STEP 1090 H=1090:00 +CPU= 20.288 22:06:55 STEP 1091 H=1091:00 +CPU= 19.768 22:07:29 STEP 1092 H=1092:00 +CPU= 33.894 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092 22:07:49 STEP 1093 H=1093:00 +CPU= 19.754 22:08:09 STEP 1094 H=1094:00 +CPU= 19.890 22:08:40 STEP 1095 H=1095:00 +CPU= 30.547 22:09:00 STEP 1096 H=1096:00 +CPU= 20.300 22:09:20 STEP 1097 H=1097:00 +CPU= 19.666 22:09:52 STEP 1098 H=1098:00 +CPU= 31.519 22:10:12 STEP 1099 H=1099:00 +CPU= 19.619 22:10:32 STEP 1100 H=1100:00 +CPU= 20.234 22:11:04 STEP 1101 H=1101:00 +CPU= 30.874 22:11:26 STEP 1102 H=1102:00 +CPU= 20.574 22:11:46 STEP 1103 H=1103:00 +CPU= 20.287 22:12:23 STEP 1104 H=1104:00 +CPU= 35.945 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001104 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001104 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001092 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_45.zip Uploading the intermediate file: upload_file_45.zip 22:12:44 STEP 1105 H=1105:00 +CPU= 19.838 22:13:04 STEP 1106 H=1106:00 +CPU= 20.267 22:13:36 STEP 1107 H=1107:00 +CPU= 30.897 22:13:58 STEP 1108 H=1108:00 +CPU= 20.703 22:14:19 STEP 1109 H=1109:00 +CPU= 20.027 22:14:51 STEP 1110 H=1110:00 +CPU= 31.509 22:15:11 STEP 1111 H=1111:00 +CPU= 20.178 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:17:45 STEP 1112 H=1112:00 +CPU= 20.803 22:18:17 STEP 1113 H=1113:00 +CPU= 30.803 22:18:38 STEP 1114 H=1114:00 +CPU= 20.552 22:18:58 STEP 1115 H=1115:00 +CPU= 19.946 22:19:32 STEP 1116 H=1116:00 +CPU= 34.070 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001116 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001116 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001116 22:19:52 STEP 1117 H=1117:00 +CPU= 19.799 22:20:13 STEP 1118 H=1118:00 +CPU= 20.304 22:20:43 STEP 1119 H=1119:00 +CPU= 29.865 22:21:03 STEP 1120 H=1120:00 +CPU= 20.362 22:21:23 STEP 1121 H=1121:00 +CPU= 19.809 22:21:55 STEP 1122 H=1122:00 +CPU= 31.649 22:22:15 STEP 1123 H=1123:00 +CPU= 19.772 22:22:37 STEP 1124 H=1124:00 +CPU= 21.051 22:23:10 STEP 1125 H=1125:00 +CPU= 31.729 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:24:18 STEP 1126 H=1126:00 +CPU= 20.409 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:26:28 STEP 1127 H=1127:00 +CPU= 20.360 22:27:04 STEP 1128 H=1128:00 +CPU= 35.673 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001128 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001128 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001116 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_46.zip Uploading the intermediate file: upload_file_46.zip 22:27:23 STEP 1129 H=1129:00 +CPU= 19.653 22:27:44 STEP 1130 H=1130:00 +CPU= 20.148 22:28:15 STEP 1131 H=1131:00 +CPU= 30.679 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1997050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0210 (argv4) batchid: 966 (argv5) wuid: 12182854 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_d8b00b9344959ed1ed7a16af55a49efd to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip ic_ancil_file: ic_ancil_12182854 ifsdata_file: ifsdata_12182854 climate_data_file: clim_data_12182854 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_93925be62ffebdba57ac843b302af8cb to: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_50410fe7985a2f73445c6a2671fc832d to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12182854.xml -- Model is restarting -- Adjusting last_iter, 1132, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml last_cpu_time: 26918 upload_file_number: 47 last_iter: 1127 last_upload: 4060800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993 The child process has been launched with process id: 54528 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 22:30:03 STEP 1128 H=1128:00 +CPU= 34.921 22:30:22 STEP 1129 H=1129:00 +CPU= 19.096 22:30:43 STEP 1130 H=1130:00 +CPU= 20.175 22:31:14 STEP 1131 H=1131:00 +CPU= 30.962 22:31:34 STEP 1132 H=1132:00 +CPU= 20.039 22:31:55 STEP 1133 H=1133:00 +CPU= 20.230 22:32:31 STEP 1134 H=1134:00 +CPU= 33.072 22:32:51 STEP 1135 H=1135:00 +CPU= 19.023 22:33:11 STEP 1136 H=1136:00 +CPU= 19.953 22:33:43 STEP 1137 H=1137:00 +CPU= 31.095 22:34:03 STEP 1138 H=1138:00 +CPU= 20.105 22:34:23 STEP 1139 H=1139:00 +CPU= 19.316 22:34:57 STEP 1140 H=1140:00 +CPU= 33.901 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001140 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001140 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001140 22:35:16 STEP 1141 H=1141:00 +CPU= 19.440 22:35:36 STEP 1142 H=1142:00 +CPU= 19.786 22:36:08 STEP 1143 H=1143:00 +CPU= 30.710 22:36:28 STEP 1144 H=1144:00 +CPU= 19.959 22:36:47 STEP 1145 H=1145:00 +CPU= 19.170 22:37:18 STEP 1146 H=1146:00 +CPU= 30.687 The child process terminated with status: 0 ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 15:48:41 | 1509499 | 22267994 | oifs_43r3_ps_0210_1997050100_123_966_12182854_0 | 3,715,200 | 24,558 | 0.0066 |
©2024 climateprediction.net