climateprediction.net home page
Task 22267994

Task 22267994

Name oifs_43r3_ps_0210_1997050100_123_966_12182854_0
Workunit 12182854
Created 22 Dec 2022, 13:00:10 UTC
Sent 22 Dec 2022, 13:02:33 UTC
Report deadline 21 Jan 2023, 13:02:33 UTC
Received 23 Dec 2022, 7:54:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1509499
Run time 7 hours 50 min 12 sec
CPU time 7 hours 36 min 31 sec
Validate state Invalid
Credit 5,201.28
Device peak FLOPS 5.06 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,413.59 MB
Peak swap size 4,978.07 MB
Peak disk usage 5,259.73 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_29.zip
Uploading the intermediate file: upload_file_29.zip
  19:32:31 STEP  721 H= 721:00 +CPU= 18.192
  19:32:50 STEP  722 H= 722:00 +CPU= 18.333
  19:33:19 STEP  723 H= 723:00 +CPU= 28.913
  19:33:39 STEP  724 H= 724:00 +CPU= 19.032
  19:33:57 STEP  725 H= 725:00 +CPU= 18.052
  19:34:28 STEP  726 H= 726:00 +CPU= 29.779
  19:34:47 STEP  727 H= 727:00 +CPU= 18.351
  19:35:06 STEP  728 H= 728:00 +CPU= 18.779
  19:35:35 STEP  729 H= 729:00 +CPU= 28.555
  19:35:53 STEP  730 H= 730:00 +CPU= 18.451
  19:36:11 STEP  731 H= 731:00 +CPU= 17.549
  19:36:42 STEP  732 H= 732:00 +CPU= 31.167
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000732
  19:37:00 STEP  733 H= 733:00 +CPU= 17.784
  19:37:18 STEP  734 H= 734:00 +CPU= 18.059
  19:37:47 STEP  735 H= 735:00 +CPU= 28.318
  19:38:05 STEP  736 H= 736:00 +CPU= 18.263
  19:38:23 STEP  737 H= 737:00 +CPU= 17.867
  19:38:54 STEP  738 H= 738:00 +CPU= 29.594
  19:39:13 STEP  739 H= 739:00 +CPU= 18.396
  19:39:31 STEP  740 H= 740:00 +CPU= 18.244
  19:39:59 STEP  741 H= 741:00 +CPU= 28.174
  19:40:18 STEP  742 H= 742:00 +CPU= 18.221
  19:40:36 STEP  743 H= 743:00 +CPU= 18.343
  19:41:11 STEP  744 H= 744:00 +CPU= 33.632
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_30.zip
Uploading the intermediate file: upload_file_30.zip
  19:41:28 STEP  745 H= 745:00 +CPU= 17.759
  19:41:47 STEP  746 H= 746:00 +CPU= 18.186
  19:42:15 STEP  747 H= 747:00 +CPU= 28.152
  19:42:33 STEP  748 H= 748:00 +CPU= 18.265
  19:42:51 STEP  749 H= 749:00 +CPU= 17.545
  19:43:20 STEP  750 H= 750:00 +CPU= 28.786
  19:43:38 STEP  751 H= 751:00 +CPU= 17.834
  19:43:56 STEP  752 H= 752:00 +CPU= 18.083
  19:44:24 STEP  753 H= 753:00 +CPU= 28.208
  19:44:42 STEP  754 H= 754:00 +CPU= 18.261
  19:45:00 STEP  755 H= 755:00 +CPU= 17.549
  19:45:31 STEP  756 H= 756:00 +CPU= 31.184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000756
  19:45:49 STEP  757 H= 757:00 +CPU= 17.780
  19:46:07 STEP  758 H= 758:00 +CPU= 18.067
  19:46:36 STEP  759 H= 759:00 +CPU= 28.344
  19:46:54 STEP  760 H= 760:00 +CPU= 18.267
  19:47:12 STEP  761 H= 761:00 +CPU= 17.499
  19:47:41 STEP  762 H= 762:00 +CPU= 28.837
  19:47:58 STEP  763 H= 763:00 +CPU= 17.674
  19:48:17 STEP  764 H= 764:00 +CPU= 18.315
  19:48:45 STEP  765 H= 765:00 +CPU= 28.204
  19:49:04 STEP  766 H= 766:00 +CPU= 18.184
  19:49:22 STEP  767 H= 767:00 +CPU= 18.018
  19:49:55 STEP  768 H= 768:00 +CPU= 33.251
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_31.zip
Uploading the intermediate file: upload_file_31.zip
  19:50:13 STEP  769 H= 769:00 +CPU= 17.767
  19:50:31 STEP  770 H= 770:00 +CPU= 18.066
  19:50:59 STEP  771 H= 771:00 +CPU= 28.261
  19:51:18 STEP  772 H= 772:00 +CPU= 18.250
  19:51:35 STEP  773 H= 773:00 +CPU= 17.538
  19:52:04 STEP  774 H= 774:00 +CPU= 28.838
  19:52:22 STEP  775 H= 775:00 +CPU= 17.784
  19:52:40 STEP  776 H= 776:00 +CPU= 18.108
  19:53:09 STEP  777 H= 777:00 +CPU= 28.206
  19:53:27 STEP  778 H= 778:00 +CPU= 18.239
  19:53:45 STEP  779 H= 779:00 +CPU= 17.541
  19:54:16 STEP  780 H= 780:00 +CPU= 31.161
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000780
  19:54:34 STEP  781 H= 781:00 +CPU= 17.747
  19:54:52 STEP  782 H= 782:00 +CPU= 18.060
  19:55:20 STEP  783 H= 783:00 +CPU= 28.390
  19:55:39 STEP  784 H= 784:00 +CPU= 18.241
  19:55:56 STEP  785 H= 785:00 +CPU= 17.509
  19:56:25 STEP  786 H= 786:00 +CPU= 28.717
  19:56:43 STEP  787 H= 787:00 +CPU= 17.660
  19:57:01 STEP  788 H= 788:00 +CPU= 18.065
  19:57:29 STEP  789 H= 789:00 +CPU= 28.147
  19:57:47 STEP  790 H= 790:00 +CPU= 18.178
  19:58:05 STEP  791 H= 791:00 +CPU= 17.994
  19:58:38 STEP  792 H= 792:00 +CPU= 32.972
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_32.zip
Uploading the intermediate file: upload_file_32.zip
  19:58:56 STEP  793 H= 793:00 +CPU= 17.744
  19:59:14 STEP  794 H= 794:00 +CPU= 18.139
  19:59:43 STEP  795 H= 795:00 +CPU= 28.190
  20:00:01 STEP  796 H= 796:00 +CPU= 18.236
  20:00:19 STEP  797 H= 797:00 +CPU= 17.557
  20:00:48 STEP  798 H= 798:00 +CPU= 28.830
  20:01:05 STEP  799 H= 799:00 +CPU= 17.761
  20:01:24 STEP  800 H= 800:00 +CPU= 18.104
  20:01:52 STEP  801 H= 801:00 +CPU= 28.194
  20:02:10 STEP  802 H= 802:00 +CPU= 18.395
  20:02:28 STEP  803 H= 803:00 +CPU= 17.540
  20:02:59 STEP  804 H= 804:00 +CPU= 31.129
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000804
  20:03:17 STEP  805 H= 805:00 +CPU= 17.758
  20:03:35 STEP  806 H= 806:00 +CPU= 18.036
  20:04:03 STEP  807 H= 807:00 +CPU= 28.210
  20:04:22 STEP  808 H= 808:00 +CPU= 18.234
  20:04:39 STEP  809 H= 809:00 +CPU= 17.544
  20:05:08 STEP  810 H= 810:00 +CPU= 28.742
  20:05:26 STEP  811 H= 811:00 +CPU= 17.668
  20:05:44 STEP  812 H= 812:00 +CPU= 18.128
  20:06:12 STEP  813 H= 813:00 +CPU= 28.152
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0210
(argv4) batchid: 966
(argv5) wuid: 12182854
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d8b00b9344959ed1ed7a16af55a49efd to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip
ic_ancil_file: ic_ancil_12182854
ifsdata_file: ifsdata_12182854
climate_data_file: clim_data_12182854
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_93925be62ffebdba57ac843b302af8cb to: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_50410fe7985a2f73445c6a2671fc832d to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12182854.xml
-- Model is restarting --
Adjusting last_iter, 813, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml
last_cpu_time: 19271
upload_file_number: 33
last_iter: 791
last_upload: 2851200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993
The child process has been launched with process id: 23004
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:23004:23004] [20221222:200722:1671736042:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:07:56 STEP  792 H= 792:00 +CPU= 31.556
  20:08:14 STEP  793 H= 793:00 +CPU= 17.722
  20:08:32 STEP  794 H= 794:00 +CPU= 18.269
  20:09:00 STEP  795 H= 795:00 +CPU= 28.075
  20:09:18 STEP  796 H= 796:00 +CPU= 18.269
  20:09:37 STEP  797 H= 797:00 +CPU= 18.104
  20:10:06 STEP  798 H= 798:00 +CPU= 28.998
  20:10:24 STEP  799 H= 799:00 +CPU= 18.057
  20:10:43 STEP  800 H= 800:00 +CPU= 18.504
  20:11:11 STEP  801 H= 801:00 +CPU= 28.167
  20:11:30 STEP  802 H= 802:00 +CPU= 18.310
  20:11:48 STEP  803 H= 803:00 +CPU= 17.810
  20:12:19 STEP  804 H= 804:00 +CPU= 31.383
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000804
  20:12:37 STEP  805 H= 805:00 +CPU= 17.832
  20:12:56 STEP  806 H= 806:00 +CPU= 18.391
  20:13:24 STEP  807 H= 807:00 +CPU= 28.310
  20:13:43 STEP  808 H= 808:00 +CPU= 18.506
  20:14:01 STEP  809 H= 809:00 +CPU= 17.862
  20:14:30 STEP  810 H= 810:00 +CPU= 28.941
  20:14:48 STEP  811 H= 811:00 +CPU= 17.787
  20:15:06 STEP  812 H= 812:00 +CPU= 18.297
  20:15:34 STEP  813 H= 813:00 +CPU= 28.098
  20:15:53 STEP  814 H= 814:00 +CPU= 18.277
  20:16:11 STEP  815 H= 815:00 +CPU= 18.284
  20:16:44 STEP  816 H= 816:00 +CPU= 32.926
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_33.zip
Uploading the intermediate file: upload_file_33.zip
  20:17:02 STEP  817 H= 817:00 +CPU= 17.695
  20:17:20 STEP  818 H= 818:00 +CPU= 18.357
  20:17:48 STEP  819 H= 819:00 +CPU= 28.027
  20:18:07 STEP  820 H= 820:00 +CPU= 18.284
  20:18:25 STEP  821 H= 821:00 +CPU= 17.924
  20:18:54 STEP  822 H= 822:00 +CPU= 28.794
  20:19:12 STEP  823 H= 823:00 +CPU= 17.957
  20:19:31 STEP  824 H= 824:00 +CPU= 18.428
  20:19:59 STEP  825 H= 825:00 +CPU= 28.346
  20:20:17 STEP  826 H= 826:00 +CPU= 18.326
  20:20:35 STEP  827 H= 827:00 +CPU= 17.895
  20:21:07 STEP  828 H= 828:00 +CPU= 31.379
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000828
  20:21:25 STEP  829 H= 829:00 +CPU= 17.937
  20:21:44 STEP  830 H= 830:00 +CPU= 18.469
  20:22:12 STEP  831 H= 831:00 +CPU= 28.283
  20:22:31 STEP  832 H= 832:00 +CPU= 18.521
  20:22:49 STEP  833 H= 833:00 +CPU= 17.968
  20:23:18 STEP  834 H= 834:00 +CPU= 28.917
  20:23:36 STEP  835 H= 835:00 +CPU= 17.820
  20:23:55 STEP  836 H= 836:00 +CPU= 18.357
  20:24:23 STEP  837 H= 837:00 +CPU= 28.306
  20:24:41 STEP  838 H= 838:00 +CPU= 18.350
  20:25:00 STEP  839 H= 839:00 +CPU= 18.328
  20:25:33 STEP  840 H= 840:00 +CPU= 33.100
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_34.zip
Uploading the intermediate file: upload_file_34.zip
  20:25:51 STEP  841 H= 841:00 +CPU= 17.748
  20:26:10 STEP  842 H= 842:00 +CPU= 18.469
  20:26:38 STEP  843 H= 843:00 +CPU= 28.129
  20:26:57 STEP  844 H= 844:00 +CPU= 18.501
  20:27:15 STEP  845 H= 845:00 +CPU= 17.817
  20:27:44 STEP  846 H= 846:00 +CPU= 29.020
  20:28:02 STEP  847 H= 847:00 +CPU= 17.928
  20:28:20 STEP  848 H= 848:00 +CPU= 18.411
  20:28:49 STEP  849 H= 849:00 +CPU= 28.285
  20:29:07 STEP  850 H= 850:00 +CPU= 18.466
  20:29:25 STEP  851 H= 851:00 +CPU= 17.804
  20:29:57 STEP  852 H= 852:00 +CPU= 31.130
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000852
  20:30:15 STEP  853 H= 853:00 +CPU= 17.922
  20:30:33 STEP  854 H= 854:00 +CPU= 18.428
  20:31:01 STEP  855 H= 855:00 +CPU= 28.231
  20:31:20 STEP  856 H= 856:00 +CPU= 18.366
  20:31:38 STEP  857 H= 857:00 +CPU= 17.858
  20:32:07 STEP  858 H= 858:00 +CPU= 28.855
  20:32:25 STEP  859 H= 859:00 +CPU= 17.863
  20:32:43 STEP  860 H= 860:00 +CPU= 18.301
  20:33:12 STEP  861 H= 861:00 +CPU= 28.472
  20:33:30 STEP  862 H= 862:00 +CPU= 18.235
  20:33:49 STEP  863 H= 863:00 +CPU= 18.244
  20:34:22 STEP  864 H= 864:00 +CPU= 32.925
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_35.zip
Uploading the intermediate file: upload_file_35.zip
  20:34:39 STEP  865 H= 865:00 +CPU= 17.698
  20:34:58 STEP  866 H= 866:00 +CPU= 18.195
  20:35:26 STEP  867 H= 867:00 +CPU= 28.134
  20:35:44 STEP  868 H= 868:00 +CPU= 18.262
  20:36:02 STEP  869 H= 869:00 +CPU= 17.736
  20:36:31 STEP  870 H= 870:00 +CPU= 28.837
  20:36:49 STEP  871 H= 871:00 +CPU= 17.742
  20:37:07 STEP  872 H= 872:00 +CPU= 18.283
  20:37:35 STEP  873 H= 873:00 +CPU= 28.213
  20:37:54 STEP  874 H= 874:00 +CPU= 18.287
  20:38:11 STEP  875 H= 875:00 +CPU= 17.801
  20:38:43 STEP  876 H= 876:00 +CPU= 31.128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000876
  20:39:01 STEP  877 H= 877:00 +CPU= 18.060
  20:39:20 STEP  878 H= 878:00 +CPU= 18.539
  20:39:48 STEP  879 H= 879:00 +CPU= 28.159
  20:40:06 STEP  880 H= 880:00 +CPU= 18.333
  20:40:24 STEP  881 H= 881:00 +CPU= 17.778
  20:40:53 STEP  882 H= 882:00 +CPU= 28.927
  20:41:11 STEP  883 H= 883:00 +CPU= 17.739
  20:41:30 STEP  884 H= 884:00 +CPU= 18.278
  20:41:58 STEP  885 H= 885:00 +CPU= 28.192
  20:42:16 STEP  886 H= 886:00 +CPU= 18.265
  20:42:35 STEP  887 H= 887:00 +CPU= 18.289
  20:43:08 STEP  888 H= 888:00 +CPU= 32.990
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_36.zip
Uploading the intermediate file: upload_file_36.zip
  20:43:26 STEP  889 H= 889:00 +CPU= 17.927
  20:43:44 STEP  890 H= 890:00 +CPU= 18.321
  20:44:12 STEP  891 H= 891:00 +CPU= 28.242
  20:44:31 STEP  892 H= 892:00 +CPU= 18.322
  20:44:49 STEP  893 H= 893:00 +CPU= 17.789
  20:45:18 STEP  894 H= 894:00 +CPU= 29.019
  20:45:36 STEP  895 H= 895:00 +CPU= 17.797
  20:45:54 STEP  896 H= 896:00 +CPU= 18.347
  20:46:22 STEP  897 H= 897:00 +CPU= 28.251
  20:46:41 STEP  898 H= 898:00 +CPU= 18.346
  20:46:59 STEP  899 H= 899:00 +CPU= 17.840
  20:47:30 STEP  900 H= 900:00 +CPU= 31.142
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000900
  20:47:48 STEP  901 H= 901:00 +CPU= 17.764
  20:48:06 STEP  902 H= 902:00 +CPU= 18.384
  20:48:35 STEP  903 H= 903:00 +CPU= 28.149
  20:48:53 STEP  904 H= 904:00 +CPU= 18.364
  20:49:11 STEP  905 H= 905:00 +CPU= 17.831
  20:49:40 STEP  906 H= 906:00 +CPU= 28.858
  20:49:58 STEP  907 H= 907:00 +CPU= 17.829
  20:50:16 STEP  908 H= 908:00 +CPU= 18.299
  20:50:44 STEP  909 H= 909:00 +CPU= 28.167
  20:51:03 STEP  910 H= 910:00 +CPU= 18.815
  20:51:22 STEP  911 H= 911:00 +CPU= 18.302
  20:51:55 STEP  912 H= 912:00 +CPU= 33.136
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_37.zip
Uploading the intermediate file: upload_file_37.zip
  20:52:13 STEP  913 H= 913:00 +CPU= 17.822
  20:52:31 STEP  914 H= 914:00 +CPU= 17.792
  20:52:58 STEP  915 H= 915:00 +CPU= 26.901
  20:53:15 STEP  916 H= 916:00 +CPU= 17.292
  20:53:32 STEP  917 H= 917:00 +CPU= 17.243
  20:54:01 STEP  918 H= 918:00 +CPU= 28.827
  20:54:19 STEP  919 H= 919:00 +CPU= 17.800
  20:54:38 STEP  920 H= 920:00 +CPU= 18.541
  20:55:06 STEP  921 H= 921:00 +CPU= 28.287
  20:55:25 STEP  922 H= 922:00 +CPU= 18.346
  20:55:43 STEP  923 H= 923:00 +CPU= 17.828
  20:56:14 STEP  924 H= 924:00 +CPU= 31.148
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000924
  20:56:32 STEP  925 H= 925:00 +CPU= 17.777
  20:56:50 STEP  926 H= 926:00 +CPU= 18.302
  20:57:18 STEP  927 H= 927:00 +CPU= 28.179
  20:57:37 STEP  928 H= 928:00 +CPU= 18.256
  20:57:54 STEP  929 H= 929:00 +CPU= 17.742
  20:58:23 STEP  930 H= 930:00 +CPU= 28.717
  20:58:41 STEP  931 H= 931:00 +CPU= 17.783
  20:58:59 STEP  932 H= 932:00 +CPU= 18.335
  20:59:28 STEP  933 H= 933:00 +CPU= 28.159
  20:59:46 STEP  934 H= 934:00 +CPU= 18.335
  21:00:05 STEP  935 H= 935:00 +CPU= 18.333
  21:00:38 STEP  936 H= 936:00 +CPU= 32.999
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_38.zip
Uploading the intermediate file: upload_file_38.zip
  21:00:56 STEP  937 H= 937:00 +CPU= 17.818
  21:01:14 STEP  938 H= 938:00 +CPU= 18.354
  21:01:42 STEP  939 H= 939:00 +CPU= 28.196
  21:02:01 STEP  940 H= 940:00 +CPU= 18.372
  21:02:19 STEP  941 H= 941:00 +CPU= 17.871
  21:02:48 STEP  942 H= 942:00 +CPU= 28.814
  21:03:06 STEP  943 H= 943:00 +CPU= 17.810
  21:03:24 STEP  944 H= 944:00 +CPU= 18.341
  21:03:52 STEP  945 H= 945:00 +CPU= 28.362
  21:04:11 STEP  946 H= 946:00 +CPU= 18.369
  21:04:29 STEP  947 H= 947:00 +CPU= 17.800
  21:05:00 STEP  948 H= 948:00 +CPU= 31.193
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000948
  21:05:18 STEP  949 H= 949:00 +CPU= 17.841
  21:05:36 STEP  950 H= 950:00 +CPU= 18.354
  21:06:05 STEP  951 H= 951:00 +CPU= 28.266
  21:06:23 STEP  952 H= 952:00 +CPU= 18.365
  21:06:41 STEP  953 H= 953:00 +CPU= 17.806
  21:07:10 STEP  954 H= 954:00 +CPU= 28.767
  21:07:28 STEP  955 H= 955:00 +CPU= 17.776
  21:07:46 STEP  956 H= 956:00 +CPU= 18.267
  21:08:14 STEP  957 H= 957:00 +CPU= 28.174
  21:08:33 STEP  958 H= 958:00 +CPU= 18.300
  21:08:51 STEP  959 H= 959:00 +CPU= 18.251
  21:09:24 STEP  960 H= 960:00 +CPU= 32.855
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_39.zip
Uploading the intermediate file: upload_file_39.zip
  21:09:42 STEP  961 H= 961:00 +CPU= 17.758
  21:10:00 STEP  962 H= 962:00 +CPU= 18.241
  21:10:28 STEP  963 H= 963:00 +CPU= 28.110
  21:10:47 STEP  964 H= 964:00 +CPU= 18.226
  21:11:04 STEP  965 H= 965:00 +CPU= 17.732
  21:11:33 STEP  966 H= 966:00 +CPU= 28.776
  21:11:51 STEP  967 H= 967:00 +CPU= 17.802
  21:12:10 STEP  968 H= 968:00 +CPU= 18.352
  21:12:38 STEP  969 H= 969:00 +CPU= 28.175
  21:12:56 STEP  970 H= 970:00 +CPU= 18.312
  21:13:14 STEP  971 H= 971:00 +CPU= 17.758
  21:13:45 STEP  972 H= 972:00 +CPU= 31.244
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000972
  21:14:03 STEP  973 H= 973:00 +CPU= 17.784
  21:14:22 STEP  974 H= 974:00 +CPU= 18.271
  21:14:50 STEP  975 H= 975:00 +CPU= 28.316
  21:15:09 STEP  976 H= 976:00 +CPU= 18.541
  21:15:27 STEP  977 H= 977:00 +CPU= 17.753
  21:15:56 STEP  978 H= 978:00 +CPU= 29.053
  21:16:14 STEP  979 H= 979:00 +CPU= 17.931
  21:16:32 STEP  980 H= 980:00 +CPU= 18.283
  21:17:01 STEP  981 H= 981:00 +CPU= 28.352
  21:17:19 STEP  982 H= 982:00 +CPU= 18.291
  21:17:38 STEP  983 H= 983:00 +CPU= 18.398
  21:18:11 STEP  984 H= 984:00 +CPU= 33.195
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_40.zip
Uploading the intermediate file: upload_file_40.zip
  21:18:29 STEP  985 H= 985:00 +CPU= 17.912
  21:18:48 STEP  986 H= 986:00 +CPU= 18.220
  21:19:16 STEP  987 H= 987:00 +CPU= 28.336
  21:19:34 STEP  988 H= 988:00 +CPU= 18.295
  21:19:52 STEP  989 H= 989:00 +CPU= 17.778
  21:20:21 STEP  990 H= 990:00 +CPU= 28.838
  21:20:39 STEP  991 H= 991:00 +CPU= 17.815
  21:20:58 STEP  992 H= 992:00 +CPU= 18.406
  21:21:26 STEP  993 H= 993:00 +CPU= 28.302
  21:21:44 STEP  994 H= 994:00 +CPU= 18.362
  21:22:02 STEP  995 H= 995:00 +CPU= 17.952
  21:22:34 STEP  996 H= 996:00 +CPU= 31.501
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000996
  21:22:52 STEP  997 H= 997:00 +CPU= 17.946
  21:23:11 STEP  998 H= 998:00 +CPU= 18.376
  21:23:39 STEP  999 H= 999:00 +CPU= 28.309
  21:23:58 STEP 1000 H=1000:00 +CPU= 18.429
  21:24:15 STEP 1001 H=1001:00 +CPU= 17.772
  21:24:44 STEP 1002 H=1002:00 +CPU= 29.030
  21:25:03 STEP 1003 H=1003:00 +CPU= 18.027
  21:25:21 STEP 1004 H=1004:00 +CPU= 18.424
  21:25:50 STEP 1005 H=1005:00 +CPU= 28.251
  21:26:08 STEP 1006 H=1006:00 +CPU= 18.298
  21:26:26 STEP 1007 H=1007:00 +CPU= 18.279
  21:26:59 STEP 1008 H=1008:00 +CPU= 33.041
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_41.zip
Uploading the intermediate file: upload_file_41.zip
  21:27:17 STEP 1009 H=1009:00 +CPU= 17.826
  21:27:36 STEP 1010 H=1010:00 +CPU= 18.262
  21:28:04 STEP 1011 H=1011:00 +CPU= 28.422
  21:28:22 STEP 1012 H=1012:00 +CPU= 18.383
  21:28:40 STEP 1013 H=1013:00 +CPU= 17.715
  21:29:09 STEP 1014 H=1014:00 +CPU= 28.912
  21:29:27 STEP 1015 H=1015:00 +CPU= 17.838
  21:29:46 STEP 1016 H=1016:00 +CPU= 18.330
  21:30:14 STEP 1017 H=1017:00 +CPU= 28.293
  21:30:32 STEP 1018 H=1018:00 +CPU= 18.369
  21:30:50 STEP 1019 H=1019:00 +CPU= 17.814
  21:31:21 STEP 1020 H=1020:00 +CPU= 31.204
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001020
  21:31:39 STEP 1021 H=1021:00 +CPU= 17.863
  21:31:58 STEP 1022 H=1022:00 +CPU= 18.307
  21:32:26 STEP 1023 H=1023:00 +CPU= 28.316
  21:32:45 STEP 1024 H=1024:00 +CPU= 18.360
  21:33:02 STEP 1025 H=1025:00 +CPU= 17.810
  21:33:31 STEP 1026 H=1026:00 +CPU= 28.812
  21:33:49 STEP 1027 H=1027:00 +CPU= 17.744
  21:34:07 STEP 1028 H=1028:00 +CPU= 18.263
  21:34:36 STEP 1029 H=1029:00 +CPU= 28.371
  21:34:56 STEP 1030 H=1030:00 +CPU= 19.427
  21:35:15 STEP 1031 H=1031:00 +CPU= 18.697
  21:35:49 STEP 1032 H=1032:00 +CPU= 33.170
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_42.zip
Uploading the intermediate file: upload_file_42.zip
  21:36:07 STEP 1033 H=1033:00 +CPU= 17.897
Uploading trickle at timestep: 3715200
  21:36:26 STEP 1034 H=1034:00 +CPU= 18.344
  21:36:54 STEP 1035 H=1035:00 +CPU= 28.476
  21:37:14 STEP 1036 H=1036:00 +CPU= 18.957
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:37:43 STEP 1037 H=1037:00 +CPU= 18.341
  21:38:13 STEP 1038 H=1038:00 +CPU= 29.159
  21:38:31 STEP 1039 H=1039:00 +CPU= 17.874
  21:38:49 STEP 1040 H=1040:00 +CPU= 18.186
  21:39:18 STEP 1041 H=1041:00 +CPU= 28.497
  21:39:36 STEP 1042 H=1042:00 +CPU= 18.384
  21:39:54 STEP 1043 H=1043:00 +CPU= 17.696
  21:40:26 STEP 1044 H=1044:00 +CPU= 31.432
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001044
  21:40:44 STEP 1045 H=1045:00 +CPU= 18.034
  21:41:08 STEP 1046 H=1046:00 +CPU= 19.782
  21:41:40 STEP 1047 H=1047:00 +CPU= 31.060
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:49:00 STEP 1048 H=1048:00 +CPU= 23.654
  21:49:20 STEP 1049 H=1049:00 +CPU= 19.795
  21:49:52 STEP 1050 H=1050:00 +CPU= 31.463
  21:50:12 STEP 1051 H=1051:00 +CPU= 19.810
  21:50:35 STEP 1052 H=1052:00 +CPU= 20.960
  21:51:06 STEP 1053 H=1053:00 +CPU= 30.832
  21:51:27 STEP 1054 H=1054:00 +CPU= 20.217
  21:51:48 STEP 1055 H=1055:00 +CPU= 20.839
  21:52:33 STEP 1056 H=1056:00 +CPU= 37.169
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_43.zip
Uploading the intermediate file: upload_file_43.zip
  21:52:54 STEP 1057 H=1057:00 +CPU= 19.936
  21:53:16 STEP 1058 H=1058:00 +CPU= 20.469
  21:53:49 STEP 1059 H=1059:00 +CPU= 31.600
  21:54:10 STEP 1060 H=1060:00 +CPU= 20.357
  21:54:30 STEP 1061 H=1061:00 +CPU= 19.700
  21:55:03 STEP 1062 H=1062:00 +CPU= 31.923
  21:55:26 STEP 1063 H=1063:00 +CPU= 20.921
  21:55:47 STEP 1064 H=1064:00 +CPU= 20.484
  21:56:19 STEP 1065 H=1065:00 +CPU= 31.098
  21:56:40 STEP 1066 H=1066:00 +CPU= 20.535
  21:57:00 STEP 1067 H=1067:00 +CPU= 19.741
  21:57:36 STEP 1068 H=1068:00 +CPU= 34.644
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068
  21:57:57 STEP 1069 H=1069:00 +CPU= 19.966
  21:58:17 STEP 1070 H=1070:00 +CPU= 19.662
  21:58:48 STEP 1071 H=1071:00 +CPU= 30.841
  21:59:08 STEP 1072 H=1072:00 +CPU= 20.348
  21:59:28 STEP 1073 H=1073:00 +CPU= 19.785
  22:00:01 STEP 1074 H=1074:00 +CPU= 31.445
  22:00:21 STEP 1075 H=1075:00 +CPU= 19.734
  22:00:42 STEP 1076 H=1076:00 +CPU= 20.439
  22:01:14 STEP 1077 H=1077:00 +CPU= 30.924
  22:01:35 STEP 1078 H=1078:00 +CPU= 20.489
  22:01:57 STEP 1079 H=1079:00 +CPU= 20.709
  22:02:35 STEP 1080 H=1080:00 +CPU= 36.156
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_44.zip
Uploading the intermediate file: upload_file_44.zip
  22:02:56 STEP 1081 H=1081:00 +CPU= 19.818
  22:03:17 STEP 1082 H=1082:00 +CPU= 20.453
  22:03:48 STEP 1083 H=1083:00 +CPU= 31.014
  22:04:09 STEP 1084 H=1084:00 +CPU= 20.255
  22:04:30 STEP 1085 H=1085:00 +CPU= 19.924
  22:05:02 STEP 1086 H=1086:00 +CPU= 31.577
  22:05:22 STEP 1087 H=1087:00 +CPU= 19.775
  22:05:43 STEP 1088 H=1088:00 +CPU= 20.551
  22:06:14 STEP 1089 H=1089:00 +CPU= 30.603
  22:06:35 STEP 1090 H=1090:00 +CPU= 20.288
  22:06:55 STEP 1091 H=1091:00 +CPU= 19.768
  22:07:29 STEP 1092 H=1092:00 +CPU= 33.894
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
  22:07:49 STEP 1093 H=1093:00 +CPU= 19.754
  22:08:09 STEP 1094 H=1094:00 +CPU= 19.890
  22:08:40 STEP 1095 H=1095:00 +CPU= 30.547
  22:09:00 STEP 1096 H=1096:00 +CPU= 20.300
  22:09:20 STEP 1097 H=1097:00 +CPU= 19.666
  22:09:52 STEP 1098 H=1098:00 +CPU= 31.519
  22:10:12 STEP 1099 H=1099:00 +CPU= 19.619
  22:10:32 STEP 1100 H=1100:00 +CPU= 20.234
  22:11:04 STEP 1101 H=1101:00 +CPU= 30.874
  22:11:26 STEP 1102 H=1102:00 +CPU= 20.574
  22:11:46 STEP 1103 H=1103:00 +CPU= 20.287
  22:12:23 STEP 1104 H=1104:00 +CPU= 35.945
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_45.zip
Uploading the intermediate file: upload_file_45.zip
  22:12:44 STEP 1105 H=1105:00 +CPU= 19.838
  22:13:04 STEP 1106 H=1106:00 +CPU= 20.267
  22:13:36 STEP 1107 H=1107:00 +CPU= 30.897
  22:13:58 STEP 1108 H=1108:00 +CPU= 20.703
  22:14:19 STEP 1109 H=1109:00 +CPU= 20.027
  22:14:51 STEP 1110 H=1110:00 +CPU= 31.509
  22:15:11 STEP 1111 H=1111:00 +CPU= 20.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:17:45 STEP 1112 H=1112:00 +CPU= 20.803
  22:18:17 STEP 1113 H=1113:00 +CPU= 30.803
  22:18:38 STEP 1114 H=1114:00 +CPU= 20.552
  22:18:58 STEP 1115 H=1115:00 +CPU= 19.946
  22:19:32 STEP 1116 H=1116:00 +CPU= 34.070
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001116
  22:19:52 STEP 1117 H=1117:00 +CPU= 19.799
  22:20:13 STEP 1118 H=1118:00 +CPU= 20.304
  22:20:43 STEP 1119 H=1119:00 +CPU= 29.865
  22:21:03 STEP 1120 H=1120:00 +CPU= 20.362
  22:21:23 STEP 1121 H=1121:00 +CPU= 19.809
  22:21:55 STEP 1122 H=1122:00 +CPU= 31.649
  22:22:15 STEP 1123 H=1123:00 +CPU= 19.772
  22:22:37 STEP 1124 H=1124:00 +CPU= 21.051
  22:23:10 STEP 1125 H=1125:00 +CPU= 31.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:24:18 STEP 1126 H=1126:00 +CPU= 20.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:26:28 STEP 1127 H=1127:00 +CPU= 20.360
  22:27:04 STEP 1128 H=1128:00 +CPU= 35.673
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993_46.zip
Uploading the intermediate file: upload_file_46.zip
  22:27:23 STEP 1129 H=1129:00 +CPU= 19.653
  22:27:44 STEP 1130 H=1130:00 +CPU= 20.148
  22:28:15 STEP 1131 H=1131:00 +CPU= 30.679
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0210
(argv4) batchid: 966
(argv5) wuid: 12182854
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182854
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d8b00b9344959ed1ed7a16af55a49efd to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0210_1997050100_123_966_12182854.zip
ic_ancil_file: ic_ancil_12182854
ifsdata_file: ifsdata_12182854
climate_data_file: clim_data_12182854
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_93925be62ffebdba57ac843b302af8cb to: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12182854.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_50410fe7985a2f73445c6a2671fc832d to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12182854.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12182854.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12182854.xml
-- Model is restarting --
Adjusting last_iter, 1132, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12182854.xml
last_cpu_time: 26918
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0210_1997050100_123_966_12182854_0_r257630993
The child process has been launched with process id: 54528
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:54528:54528] [20221222:222926:1671744566:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:30:03 STEP 1128 H=1128:00 +CPU= 34.921
  22:30:22 STEP 1129 H=1129:00 +CPU= 19.096
  22:30:43 STEP 1130 H=1130:00 +CPU= 20.175
  22:31:14 STEP 1131 H=1131:00 +CPU= 30.962
  22:31:34 STEP 1132 H=1132:00 +CPU= 20.039
  22:31:55 STEP 1133 H=1133:00 +CPU= 20.230
  22:32:31 STEP 1134 H=1134:00 +CPU= 33.072
  22:32:51 STEP 1135 H=1135:00 +CPU= 19.023
  22:33:11 STEP 1136 H=1136:00 +CPU= 19.953
  22:33:43 STEP 1137 H=1137:00 +CPU= 31.095
  22:34:03 STEP 1138 H=1138:00 +CPU= 20.105
  22:34:23 STEP 1139 H=1139:00 +CPU= 19.316
  22:34:57 STEP 1140 H=1140:00 +CPU= 33.901
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001140
  22:35:16 STEP 1141 H=1141:00 +CPU= 19.440
  22:35:36 STEP 1142 H=1142:00 +CPU= 19.786
  22:36:08 STEP 1143 H=1143:00 +CPU= 30.710
  22:36:28 STEP 1144 H=1144:00 +CPU= 19.959
  22:36:47 STEP 1145 H=1145:00 +CPU= 19.170
  22:37:18 STEP 1146 H=1146:00 +CPU= 30.687
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:48:41 1509499 22267994 oifs_43r3_ps_0210_1997050100_123_966_12182854_0 3,715,200 24,558 0.0066


©2024 climateprediction.net