climateprediction.net home page
Task 22267698

Task 22267698

Name oifs_43r3_ps_0944_1996050100_123_965_12182588_0
Workunit 12182588
Created 22 Dec 2022, 12:17:43 UTC
Sent 22 Dec 2022, 12:51:37 UTC
Report deadline 21 Jan 2023, 12:51:37 UTC
Received 23 Dec 2022, 2:39:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1534452
Run time 12 hours 40 min
CPU time 12 hours 13 min 23 sec
Validate state Invalid
Credit 5,201.28
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,404.10 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,019.89 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
us model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182588.xml
last_cpu_time: 27560
upload_file_number: 33
last_iter: 791
last_upload: 2851200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894
The child process has been launched with process id: 15689
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:15689:15689] [20221222:215734:1671742654:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:59:21 STEP  792 H= 792:00 +CPU= 46.445
  21:59:43 STEP  793 H= 793:00 +CPU= 22.277
  22:00:10 STEP  794 H= 794:00 +CPU= 26.585
  22:00:47 STEP  795 H= 795:00 +CPU= 37.708
  22:01:15 STEP  796 H= 796:00 +CPU= 27.241
  22:01:42 STEP  797 H= 797:00 +CPU= 27.006
  22:02:21 STEP  798 H= 798:00 +CPU= 39.098
  22:02:44 STEP  799 H= 799:00 +CPU= 23.068
  22:03:03 STEP  800 H= 800:00 +CPU= 18.699
  22:03:37 STEP  801 H= 801:00 +CPU= 34.286
  22:04:02 STEP  802 H= 802:00 +CPU= 25.166
  22:04:26 STEP  803 H= 803:00 +CPU= 23.663
  22:05:09 STEP  804 H= 804:00 +CPU= 42.812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000804
  22:05:35 STEP  805 H= 805:00 +CPU= 25.813
  22:06:06 STEP  806 H= 806:00 +CPU= 30.724
  22:07:01 STEP  807 H= 807:00 +CPU= 53.011
  22:07:42 STEP  808 H= 808:00 +CPU= 37.358
  22:08:13 STEP  809 H= 809:00 +CPU= 30.743
  22:09:00 STEP  810 H= 810:00 +CPU= 45.490
  22:09:29 STEP  811 H= 811:00 +CPU= 29.214
  22:09:59 STEP  812 H= 812:00 +CPU= 29.341
  22:10:44 STEP  813 H= 813:00 +CPU= 43.408
  22:11:16 STEP  814 H= 814:00 +CPU= 29.659
  22:11:48 STEP  815 H= 815:00 +CPU= 30.099
  22:12:47 STEP  816 H= 816:00 +CPU= 50.138
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_33.zip
Uploading the intermediate file: upload_file_33.zip
  22:13:17 STEP  817 H= 817:00 +CPU= 28.440
  22:13:54 STEP  818 H= 818:00 +CPU= 36.033
  22:14:48 STEP  819 H= 819:00 +CPU= 53.312
  22:15:19 STEP  820 H= 820:00 +CPU= 30.732
  22:15:52 STEP  821 H= 821:00 +CPU= 32.810
  22:16:43 STEP  822 H= 822:00 +CPU= 49.701
  22:17:13 STEP  823 H= 823:00 +CPU= 28.737
  22:17:43 STEP  824 H= 824:00 +CPU= 29.484
  22:18:26 STEP  825 H= 825:00 +CPU= 43.009
  22:18:56 STEP  826 H= 826:00 +CPU= 29.566
  22:19:25 STEP  827 H= 827:00 +CPU= 28.995
  22:20:14 STEP  828 H= 828:00 +CPU= 48.254
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000828
  22:20:43 STEP  829 H= 829:00 +CPU= 28.160
  22:21:13 STEP  830 H= 830:00 +CPU= 29.826
  22:21:57 STEP  831 H= 831:00 +CPU= 43.990
  22:22:32 STEP  832 H= 832:00 +CPU= 31.619
  22:23:07 STEP  833 H= 833:00 +CPU= 32.045
  22:24:03 STEP  834 H= 834:00 +CPU= 55.251
  22:24:37 STEP  835 H= 835:00 +CPU= 32.735
  22:25:16 STEP  836 H= 836:00 +CPU= 37.039
  22:26:01 STEP  837 H= 837:00 +CPU= 44.712
  22:26:32 STEP  838 H= 838:00 +CPU= 28.624
  22:27:04 STEP  839 H= 839:00 +CPU= 30.459
  22:28:04 STEP  840 H= 840:00 +CPU= 50.965
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_34.zip
Uploading the intermediate file: upload_file_34.zip
  22:28:33 STEP  841 H= 841:00 +CPU= 29.029
  22:29:03 STEP  842 H= 842:00 +CPU= 29.781
  22:29:49 STEP  843 H= 843:00 +CPU= 44.691
  22:30:19 STEP  844 H= 844:00 +CPU= 29.778
  22:30:47 STEP  845 H= 845:00 +CPU= 27.908
  22:31:36 STEP  846 H= 846:00 +CPU= 46.640
  22:32:11 STEP  847 H= 847:00 +CPU= 32.120
  22:32:45 STEP  848 H= 848:00 +CPU= 33.559
  22:33:29 STEP  849 H= 849:00 +CPU= 43.611
  22:33:59 STEP  850 H= 850:00 +CPU= 29.359
  22:34:28 STEP  851 H= 851:00 +CPU= 29.391
  22:35:20 STEP  852 H= 852:00 +CPU= 48.358
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000852
  22:35:50 STEP  853 H= 853:00 +CPU= 28.609
  22:36:21 STEP  854 H= 854:00 +CPU= 30.056
  22:37:06 STEP  855 H= 855:00 +CPU= 44.148
  22:37:37 STEP  856 H= 856:00 +CPU= 30.391
  22:38:06 STEP  857 H= 857:00 +CPU= 28.827
  22:38:53 STEP  858 H= 858:00 +CPU= 46.296
  22:39:28 STEP  859 H= 859:00 +CPU= 34.116
  22:40:03 STEP  860 H= 860:00 +CPU= 33.539
  22:40:50 STEP  861 H= 861:00 +CPU= 46.927
  22:41:24 STEP  862 H= 862:00 +CPU= 33.694
  22:41:54 STEP  863 H= 863:00 +CPU= 30.295
  22:42:47 STEP  864 H= 864:00 +CPU= 50.251
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_35.zip
Uploading the intermediate file: upload_file_35.zip
  22:43:18 STEP  865 H= 865:00 +CPU= 28.280
  22:43:48 STEP  866 H= 866:00 +CPU= 29.414
  22:44:31 STEP  867 H= 867:00 +CPU= 42.542
  22:45:01 STEP  868 H= 868:00 +CPU= 30.380
  22:45:32 STEP  869 H= 869:00 +CPU= 29.457
  22:46:20 STEP  870 H= 870:00 +CPU= 45.302
  22:46:48 STEP  871 H= 871:00 +CPU= 28.069
  22:47:20 STEP  872 H= 872:00 +CPU= 29.319
  22:48:12 STEP  873 H= 873:00 +CPU= 48.628
  22:48:51 STEP  874 H= 874:00 +CPU= 34.617
  22:49:21 STEP  875 H= 875:00 +CPU= 29.659
  22:50:15 STEP  876 H= 876:00 +CPU= 49.192
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000876
  22:50:43 STEP  877 H= 877:00 +CPU= 27.841
  22:51:14 STEP  878 H= 878:00 +CPU= 29.310
  22:51:59 STEP  879 H= 879:00 +CPU= 43.854
  22:52:31 STEP  880 H= 880:00 +CPU= 29.949
  22:53:01 STEP  881 H= 881:00 +CPU= 28.609
  22:53:47 STEP  882 H= 882:00 +CPU= 44.791
  22:54:15 STEP  883 H= 883:00 +CPU= 27.722
  22:54:49 STEP  884 H= 884:00 +CPU= 32.904
  22:55:40 STEP  885 H= 885:00 +CPU= 49.623
  22:56:18 STEP  886 H= 886:00 +CPU= 36.518
  22:56:52 STEP  887 H= 887:00 +CPU= 32.881
  22:57:56 STEP  888 H= 888:00 +CPU= 61.807
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_36.zip
Uploading the intermediate file: upload_file_36.zip
  22:58:25 STEP  889 H= 889:00 +CPU= 28.185
  22:58:56 STEP  890 H= 890:00 +CPU= 30.401
  22:59:41 STEP  891 H= 891:00 +CPU= 43.554
  23:00:13 STEP  892 H= 892:00 +CPU= 29.971
  23:00:40 STEP  893 H= 893:00 +CPU= 27.256
  23:01:25 STEP  894 H= 894:00 +CPU= 44.196
  23:01:55 STEP  895 H= 895:00 +CPU= 29.440
  23:02:27 STEP  896 H= 896:00 +CPU= 29.159
  23:03:12 STEP  897 H= 897:00 +CPU= 43.268
  23:03:42 STEP  898 H= 898:00 +CPU= 30.616
  23:04:12 STEP  899 H= 899:00 +CPU= 29.849
  23:05:03 STEP  900 H= 900:00 +CPU= 50.200
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000900
  23:05:38 STEP  901 H= 901:00 +CPU= 33.138
  23:06:15 STEP  902 H= 902:00 +CPU= 35.699
  23:07:06 STEP  903 H= 903:00 +CPU= 47.345
  23:07:36 STEP  904 H= 904:00 +CPU= 29.595
  23:08:06 STEP  905 H= 905:00 +CPU= 28.917
  23:08:51 STEP  906 H= 906:00 +CPU= 44.793
  23:09:23 STEP  907 H= 907:00 +CPU= 29.277
  23:09:54 STEP  908 H= 908:00 +CPU= 29.672
  23:10:38 STEP  909 H= 909:00 +CPU= 43.896
  23:11:09 STEP  910 H= 910:00 +CPU= 29.841
  23:11:39 STEP  911 H= 911:00 +CPU= 29.560
  23:12:41 STEP  912 H= 912:00 +CPU= 56.297
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_37.zip
Uploading the intermediate file: upload_file_37.zip
  23:13:15 STEP  913 H= 913:00 +CPU= 32.325
  23:13:52 STEP  914 H= 914:00 +CPU= 36.627
  23:14:42 STEP  915 H= 915:00 +CPU= 48.879
  23:15:14 STEP  916 H= 916:00 +CPU= 31.548
  23:15:44 STEP  917 H= 917:00 +CPU= 29.798
  23:16:29 STEP  918 H= 918:00 +CPU= 44.101
  23:16:58 STEP  919 H= 919:00 +CPU= 28.189
  23:17:27 STEP  920 H= 920:00 +CPU= 29.054
  23:18:11 STEP  921 H= 921:00 +CPU= 43.359
  23:18:43 STEP  922 H= 922:00 +CPU= 29.378
  23:19:13 STEP  923 H= 923:00 +CPU= 28.823
  23:20:01 STEP  924 H= 924:00 +CPU= 47.696
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000924
  23:20:30 STEP  925 H= 925:00 +CPU= 28.235
  23:21:01 STEP  926 H= 926:00 +CPU= 30.554
  23:21:49 STEP  927 H= 927:00 +CPU= 48.379
  23:22:21 STEP  928 H= 928:00 +CPU= 31.293
  23:22:52 STEP  929 H= 929:00 +CPU= 30.765
  23:23:38 STEP  930 H= 930:00 +CPU= 46.696
  23:24:09 STEP  931 H= 931:00 +CPU= 30.244
  23:24:38 STEP  932 H= 932:00 +CPU= 29.597
  23:25:23 STEP  933 H= 933:00 +CPU= 44.103
  23:25:52 STEP  934 H= 934:00 +CPU= 29.105
  23:26:23 STEP  935 H= 935:00 +CPU= 29.994
  23:27:21 STEP  936 H= 936:00 +CPU= 51.809
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_38.zip
Uploading the intermediate file: upload_file_38.zip
  23:27:50 STEP  937 H= 937:00 +CPU= 28.155
  23:28:22 STEP  938 H= 938:00 +CPU= 30.340
  23:29:07 STEP  939 H= 939:00 +CPU= 44.225
  23:29:36 STEP  940 H= 940:00 +CPU= 29.505
  23:30:06 STEP  941 H= 941:00 +CPU= 28.565
  23:30:50 STEP  942 H= 942:00 +CPU= 44.678
  23:31:21 STEP  943 H= 943:00 +CPU= 28.687
  23:31:51 STEP  944 H= 944:00 +CPU= 29.816
  23:32:36 STEP  945 H= 945:00 +CPU= 44.689
  23:33:06 STEP  946 H= 946:00 +CPU= 29.189
  23:33:35 STEP  947 H= 947:00 +CPU= 29.062
  23:34:24 STEP  948 H= 948:00 +CPU= 48.733
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000948
  23:34:52 STEP  949 H= 949:00 +CPU= 27.942
  23:35:22 STEP  950 H= 950:00 +CPU= 29.528
  23:36:07 STEP  951 H= 951:00 +CPU= 44.507
  23:36:38 STEP  952 H= 952:00 +CPU= 29.280
  23:37:06 STEP  953 H= 953:00 +CPU= 27.928
  23:37:52 STEP  954 H= 954:00 +CPU= 45.151
  23:38:20 STEP  955 H= 955:00 +CPU= 28.059
  23:38:49 STEP  956 H= 956:00 +CPU= 29.070
  23:39:33 STEP  957 H= 957:00 +CPU= 44.645
  23:40:05 STEP  958 H= 958:00 +CPU= 29.117
  23:40:35 STEP  959 H= 959:00 +CPU= 29.770
  23:41:42 STEP  960 H= 960:00 +CPU= 51.243
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_39.zip
Uploading the intermediate file: upload_file_39.zip
  23:42:10 STEP  961 H= 961:00 +CPU= 28.318
  23:42:42 STEP  962 H= 962:00 +CPU= 29.582
  23:43:27 STEP  963 H= 963:00 +CPU= 43.609
  23:43:55 STEP  964 H= 964:00 +CPU= 28.475
  23:44:23 STEP  965 H= 965:00 +CPU= 28.127
  23:45:10 STEP  966 H= 966:00 +CPU= 44.094
  23:45:40 STEP  967 H= 967:00 +CPU= 28.783
  23:46:10 STEP  968 H= 968:00 +CPU= 29.505
  23:46:56 STEP  969 H= 969:00 +CPU= 43.516
  23:47:29 STEP  970 H= 970:00 +CPU= 30.330
  23:47:59 STEP  971 H= 971:00 +CPU= 29.192
  23:48:50 STEP  972 H= 972:00 +CPU= 48.324
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000972
  23:49:19 STEP  973 H= 973:00 +CPU= 27.495
  23:49:50 STEP  974 H= 974:00 +CPU= 29.747
  23:50:34 STEP  975 H= 975:00 +CPU= 43.342
  23:51:03 STEP  976 H= 976:00 +CPU= 29.243
  23:51:33 STEP  977 H= 977:00 +CPU= 28.817
  23:52:20 STEP  978 H= 978:00 +CPU= 45.793
  23:52:50 STEP  979 H= 979:00 +CPU= 29.263
  23:53:20 STEP  980 H= 980:00 +CPU= 29.648
  23:54:04 STEP  981 H= 981:00 +CPU= 43.639
  23:54:36 STEP  982 H= 982:00 +CPU= 29.813
  23:55:08 STEP  983 H= 983:00 +CPU= 29.842
  23:56:06 STEP  984 H= 984:00 +CPU= 50.831
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_40.zip
Uploading the intermediate file: upload_file_40.zip
  23:56:38 STEP  985 H= 985:00 +CPU= 28.042
  23:57:08 STEP  986 H= 986:00 +CPU= 29.572
  23:57:52 STEP  987 H= 987:00 +CPU= 43.454
  23:58:22 STEP  988 H= 988:00 +CPU= 29.873
  23:58:51 STEP  989 H= 989:00 +CPU= 28.933
  23:59:35 STEP  990 H= 990:00 +CPU= 43.617
  00:00:04 STEP  991 H= 991:00 +CPU= 28.811
  00:00:34 STEP  992 H= 992:00 +CPU= 29.261
  00:01:20 STEP  993 H= 993:00 +CPU= 43.324
  00:01:50 STEP  994 H= 994:00 +CPU= 29.353
  00:02:19 STEP  995 H= 995:00 +CPU= 28.864
  00:03:08 STEP  996 H= 996:00 +CPU= 48.238
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000996
  00:03:37 STEP  997 H= 997:00 +CPU= 29.086
  00:04:10 STEP  998 H= 998:00 +CPU= 29.800
  00:04:55 STEP  999 H= 999:00 +CPU= 44.324
  00:05:25 STEP 1000 H=1000:00 +CPU= 29.195
  00:05:53 STEP 1001 H=1001:00 +CPU= 27.784
  00:06:39 STEP 1002 H=1002:00 +CPU= 44.478
  00:07:10 STEP 1003 H=1003:00 +CPU= 29.368
  00:07:40 STEP 1004 H=1004:00 +CPU= 29.929
  00:08:24 STEP 1005 H=1005:00 +CPU= 43.770
  00:08:54 STEP 1006 H=1006:00 +CPU= 29.392
  00:09:26 STEP 1007 H=1007:00 +CPU= 30.943
  00:10:32 STEP 1008 H=1008:00 +CPU= 60.225
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_41.zip
Uploading the intermediate file: upload_file_41.zip
  00:11:07 STEP 1009 H=1009:00 +CPU= 33.497
  00:11:41 STEP 1010 H=1010:00 +CPU= 33.625
  00:12:31 STEP 1011 H=1011:00 +CPU= 49.818
  00:13:02 STEP 1012 H=1012:00 +CPU= 30.179
  00:13:33 STEP 1013 H=1013:00 +CPU= 28.911
  00:14:15 STEP 1014 H=1014:00 +CPU= 41.827
  00:14:46 STEP 1015 H=1015:00 +CPU= 28.992
  00:15:16 STEP 1016 H=1016:00 +CPU= 29.455
  00:16:00 STEP 1017 H=1017:00 +CPU= 42.959
  00:16:29 STEP 1018 H=1018:00 +CPU= 29.626
  00:16:59 STEP 1019 H=1019:00 +CPU= 29.557
  00:17:47 STEP 1020 H=1020:00 +CPU= 47.364
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001020
  00:18:15 STEP 1021 H=1021:00 +CPU= 27.817
  00:18:48 STEP 1022 H=1022:00 +CPU= 32.591
  00:19:42 STEP 1023 H=1023:00 +CPU= 49.440
  00:20:17 STEP 1024 H=1024:00 +CPU= 34.557
  00:20:53 STEP 1025 H=1025:00 +CPU= 34.889
  00:21:40 STEP 1026 H=1026:00 +CPU= 45.940
  00:22:10 STEP 1027 H=1027:00 +CPU= 28.668
  00:22:41 STEP 1028 H=1028:00 +CPU= 29.930
  00:23:29 STEP 1029 H=1029:00 +CPU= 44.018
  00:24:01 STEP 1030 H=1030:00 +CPU= 29.946
  00:24:32 STEP 1031 H=1031:00 +CPU= 29.637
  00:25:31 STEP 1032 H=1032:00 +CPU= 51.415
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_42.zip
Uploading the intermediate file: upload_file_42.zip
Uploading trickle at timestep: 3715200
  00:26:01 STEP 1033 H=1033:00 +CPU= 28.556
  00:26:30 STEP 1034 H=1034:00 +CPU= 29.393
  00:27:24 STEP 1035 H=1035:00 +CPU= 49.982
  00:28:04 STEP 1036 H=1036:00 +CPU= 34.941
  00:28:38 STEP 1037 H=1037:00 +CPU= 33.498
  00:29:28 STEP 1038 H=1038:00 +CPU= 50.138
  00:29:59 STEP 1039 H=1039:00 +CPU= 30.041
  00:30:29 STEP 1040 H=1040:00 +CPU= 30.132
  00:31:16 STEP 1041 H=1041:00 +CPU= 44.300
  00:31:47 STEP 1042 H=1042:00 +CPU= 29.543
  00:32:16 STEP 1043 H=1043:00 +CPU= 29.317
  00:33:05 STEP 1044 H=1044:00 +CPU= 48.760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001044
  00:33:34 STEP 1045 H=1045:00 +CPU= 27.325
  00:34:04 STEP 1046 H=1046:00 +CPU= 30.066
  00:34:51 STEP 1047 H=1047:00 +CPU= 44.408
  00:35:23 STEP 1048 H=1048:00 +CPU= 31.033
  00:35:59 STEP 1049 H=1049:00 +CPU= 33.230
  00:36:50 STEP 1050 H=1050:00 +CPU= 48.626
  00:37:24 STEP 1051 H=1051:00 +CPU= 33.909
  00:38:00 STEP 1052 H=1052:00 +CPU= 33.792
  00:38:48 STEP 1053 H=1053:00 +CPU= 46.716
  00:39:20 STEP 1054 H=1054:00 +CPU= 30.250
  00:39:50 STEP 1055 H=1055:00 +CPU= 29.765
  00:40:51 STEP 1056 H=1056:00 +CPU= 51.886
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_43.zip
Uploading the intermediate file: upload_file_43.zip
  00:41:21 STEP 1057 H=1057:00 +CPU= 28.332
  00:41:51 STEP 1058 H=1058:00 +CPU= 30.261
  00:42:37 STEP 1059 H=1059:00 +CPU= 44.911
  00:43:10 STEP 1060 H=1060:00 +CPU= 29.649
  00:43:39 STEP 1061 H=1061:00 +CPU= 28.619
  00:44:28 STEP 1062 H=1062:00 +CPU= 48.078
  00:45:01 STEP 1063 H=1063:00 +CPU= 32.958
  00:45:36 STEP 1064 H=1064:00 +CPU= 34.241
  00:46:21 STEP 1065 H=1065:00 +CPU= 44.812
  00:46:51 STEP 1066 H=1066:00 +CPU= 29.277
  00:47:19 STEP 1067 H=1067:00 +CPU= 28.227
  00:48:09 STEP 1068 H=1068:00 +CPU= 47.375
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001068
  00:48:37 STEP 1069 H=1069:00 +CPU= 27.552
  00:49:07 STEP 1070 H=1070:00 +CPU= 29.145
  00:49:51 STEP 1071 H=1071:00 +CPU= 43.823
  00:50:23 STEP 1072 H=1072:00 +CPU= 30.158
  00:50:52 STEP 1073 H=1073:00 +CPU= 28.630
  00:51:42 STEP 1074 H=1074:00 +CPU= 49.121
  00:52:15 STEP 1075 H=1075:00 +CPU= 33.193
  00:52:50 STEP 1076 H=1076:00 +CPU= 34.113
  00:53:40 STEP 1077 H=1077:00 +CPU= 49.216
  00:54:18 STEP 1078 H=1078:00 +CPU= 34.444
  00:54:48 STEP 1079 H=1079:00 +CPU= 29.787
  00:55:49 STEP 1080 H=1080:00 +CPU= 52.164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_44.zip
Uploading the intermediate file: upload_file_44.zip
  00:56:18 STEP 1081 H=1081:00 +CPU= 28.622
  00:56:49 STEP 1082 H=1082:00 +CPU= 29.539
  00:57:33 STEP 1083 H=1083:00 +CPU= 44.672
  00:58:05 STEP 1084 H=1084:00 +CPU= 29.917
  00:58:34 STEP 1085 H=1085:00 +CPU= 28.689
  00:59:17 STEP 1086 H=1086:00 +CPU= 43.186
  00:59:47 STEP 1087 H=1087:00 +CPU= 28.120
  01:00:22 STEP 1088 H=1088:00 +CPU= 32.754
  01:01:14 STEP 1089 H=1089:00 +CPU= 50.260
  01:01:47 STEP 1090 H=1090:00 +CPU= 32.411
  01:02:16 STEP 1091 H=1091:00 +CPU= 28.772
  01:03:06 STEP 1092 H=1092:00 +CPU= 48.287
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001092
  01:03:36 STEP 1093 H=1093:00 +CPU= 28.435
  01:04:07 STEP 1094 H=1094:00 +CPU= 30.012
  01:04:50 STEP 1095 H=1095:00 +CPU= 42.722
  01:05:20 STEP 1096 H=1096:00 +CPU= 29.522
  01:05:49 STEP 1097 H=1097:00 +CPU= 28.885
  01:06:37 STEP 1098 H=1098:00 +CPU= 44.930
  01:07:06 STEP 1099 H=1099:00 +CPU= 28.249
  01:07:40 STEP 1100 H=1100:00 +CPU= 33.354
  01:08:30 STEP 1101 H=1101:00 +CPU= 48.766
  01:09:06 STEP 1102 H=1102:00 +CPU= 34.365
  01:09:42 STEP 1103 H=1103:00 +CPU= 35.792
  01:10:44 STEP 1104 H=1104:00 +CPU= 53.987
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_45.zip
Uploading the intermediate file: upload_file_45.zip
  01:11:13 STEP 1105 H=1105:00 +CPU= 28.120
  01:11:43 STEP 1106 H=1106:00 +CPU= 29.648
  01:12:28 STEP 1107 H=1107:00 +CPU= 44.185
  01:13:00 STEP 1108 H=1108:00 +CPU= 29.797
  01:13:30 STEP 1109 H=1109:00 +CPU= 29.325
  01:14:15 STEP 1110 H=1110:00 +CPU= 43.421
  01:14:44 STEP 1111 H=1111:00 +CPU= 27.880
  01:15:16 STEP 1112 H=1112:00 +CPU= 28.981
  01:16:02 STEP 1113 H=1113:00 +CPU= 45.236
  01:16:38 STEP 1114 H=1114:00 +CPU= 33.687
  01:17:10 STEP 1115 H=1115:00 +CPU= 31.826
  01:18:03 STEP 1116 H=1116:00 +CPU= 50.437
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001116
  01:18:31 STEP 1117 H=1117:00 +CPU= 28.256
  01:19:01 STEP 1118 H=1118:00 +CPU= 29.408
  01:19:49 STEP 1119 H=1119:00 +CPU= 43.822
  01:20:23 STEP 1120 H=1120:00 +CPU= 29.874
  01:20:55 STEP 1121 H=1121:00 +CPU= 29.290
  01:21:40 STEP 1122 H=1122:00 +CPU= 44.535
  01:22:09 STEP 1123 H=1123:00 +CPU= 28.344
  01:22:38 STEP 1124 H=1124:00 +CPU= 28.931
  01:23:24 STEP 1125 H=1125:00 +CPU= 45.252
  01:23:57 STEP 1126 H=1126:00 +CPU= 33.026
  01:24:32 STEP 1127 H=1127:00 +CPU= 34.397
  01:25:40 STEP 1128 H=1128:00 +CPU= 59.047
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_46.zip
Uploading the intermediate file: upload_file_46.zip
  01:26:12 STEP 1129 H=1129:00 +CPU= 31.924
  01:26:45 STEP 1130 H=1130:00 +CPU= 30.423
  01:27:31 STEP 1131 H=1131:00 +CPU= 44.317
  01:28:01 STEP 1132 H=1132:00 +CPU= 30.295
  01:28:31 STEP 1133 H=1133:00 +CPU= 29.212
  01:29:16 STEP 1134 H=1134:00 +CPU= 44.909
  01:29:47 STEP 1135 H=1135:00 +CPU= 29.163
  01:30:21 STEP 1136 H=1136:00 +CPU= 30.600
  01:31:05 STEP 1137 H=1137:00 +CPU= 44.065
  01:31:35 STEP 1138 H=1138:00 +CPU= 30.498
  01:32:06 STEP 1139 H=1139:00 +CPU= 29.256
  01:32:54 STEP 1140 H=1140:00 +CPU= 47.794
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001140
  01:33:24 STEP 1141 H=1141:00 +CPU= 29.311
  01:33:55 STEP 1142 H=1142:00 +CPU= 30.794
  01:34:41 STEP 1143 H=1143:00 +CPU= 44.016
  01:35:15 STEP 1144 H=1144:00 +CPU= 30.245
  01:35:46 STEP 1145 H=1145:00 +CPU= 29.305
  01:36:34 STEP 1146 H=1146:00 +CPU= 44.930
  01:37:04 STEP 1147 H=1147:00 +CPU= 29.443
  01:37:34 STEP 1148 H=1148:00 +CPU= 30.326
  01:38:19 STEP 1149 H=1149:00 +CPU= 44.207
  01:38:49 STEP 1150 H=1150:00 +CPU= 29.684
  01:39:20 STEP 1151 H=1151:00 +CPU= 30.358
  01:40:16 STEP 1152 H=1152:00 +CPU= 51.875
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_47.zip
Uploading the intermediate file: upload_file_47.zip
  01:40:46 STEP 1153 H=1153:00 +CPU= 29.664
  01:41:19 STEP 1154 H=1154:00 +CPU= 30.602
  01:42:04 STEP 1155 H=1155:00 +CPU= 43.902
  01:42:35 STEP 1156 H=1156:00 +CPU= 30.065
  01:43:07 STEP 1157 H=1157:00 +CPU= 28.685
  01:43:54 STEP 1158 H=1158:00 +CPU= 45.769
  01:44:25 STEP 1159 H=1159:00 +CPU= 29.702
  01:44:57 STEP 1160 H=1160:00 +CPU= 30.319
  01:45:42 STEP 1161 H=1161:00 +CPU= 45.275
  01:46:12 STEP 1162 H=1162:00 +CPU= 30.049
  01:46:42 STEP 1163 H=1163:00 +CPU= 29.297
  01:47:30 STEP 1164 H=1164:00 +CPU= 48.169
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001164
  01:48:01 STEP 1165 H=1165:00 +CPU= 29.306
  01:48:32 STEP 1166 H=1166:00 +CPU= 30.707
  01:49:19 STEP 1167 H=1167:00 +CPU= 44.231
  01:49:51 STEP 1168 H=1168:00 +CPU= 30.065
  01:50:21 STEP 1169 H=1169:00 +CPU= 29.684
  01:51:08 STEP 1170 H=1170:00 +CPU= 46.274
  01:51:37 STEP 1171 H=1171:00 +CPU= 28.958
  01:52:08 STEP 1172 H=1172:00 +CPU= 30.264
  01:52:54 STEP 1173 H=1173:00 +CPU= 43.745
  01:53:26 STEP 1174 H=1174:00 +CPU= 29.544
  01:53:56 STEP 1175 H=1175:00 +CPU= 29.706
  01:55:00 STEP 1176 H=1176:00 +CPU= 51.999
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_48.zip
Uploading the intermediate file: upload_file_48.zip
  01:55:30 STEP 1177 H=1177:00 +CPU= 29.195
  01:56:01 STEP 1178 H=1178:00 +CPU= 30.167
  01:56:47 STEP 1179 H=1179:00 +CPU= 44.869
  01:57:18 STEP 1180 H=1180:00 +CPU= 30.970
  01:57:48 STEP 1181 H=1181:00 +CPU= 28.624
  01:58:35 STEP 1182 H=1182:00 +CPU= 45.522
  01:59:05 STEP 1183 H=1183:00 +CPU= 30.024
  01:59:36 STEP 1184 H=1184:00 +CPU= 30.689
  02:00:22 STEP 1185 H=1185:00 +CPU= 45.137
  02:00:53 STEP 1186 H=1186:00 +CPU= 30.176
  02:01:24 STEP 1187 H=1187:00 +CPU= 29.946
  02:02:13 STEP 1188 H=1188:00 +CPU= 48.793
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001188
  02:02:45 STEP 1189 H=1189:00 +CPU= 29.309
  02:03:15 STEP 1190 H=1190:00 +CPU= 29.820
  02:04:02 STEP 1191 H=1191:00 +CPU= 44.843
  02:04:34 STEP 1192 H=1192:00 +CPU= 30.179
  02:05:04 STEP 1193 H=1193:00 +CPU= 29.679
  02:05:51 STEP 1194 H=1194:00 +CPU= 45.181
  02:06:21 STEP 1195 H=1195:00 +CPU= 29.471
  02:06:52 STEP 1196 H=1196:00 +CPU= 30.152
  02:07:36 STEP 1197 H=1197:00 +CPU= 44.498
  02:08:07 STEP 1198 H=1198:00 +CPU= 29.869
  02:08:37 STEP 1199 H=1199:00 +CPU= 30.058
  02:09:34 STEP 1200 H=1200:00 +CPU= 51.953
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_49.zip
Uploading the intermediate file: upload_file_49.zip
  02:10:05 STEP 1201 H=1201:00 +CPU= 29.039
  02:10:35 STEP 1202 H=1202:00 +CPU= 29.346
  02:11:19 STEP 1203 H=1203:00 +CPU= 44.510
  02:11:50 STEP 1204 H=1204:00 +CPU= 30.581
  02:12:19 STEP 1205 H=1205:00 +CPU= 27.876
  02:13:05 STEP 1206 H=1206:00 +CPU= 45.335
  02:13:35 STEP 1207 H=1207:00 +CPU= 29.165
  02:14:05 STEP 1208 H=1208:00 +CPU= 30.022
  02:14:49 STEP 1209 H=1209:00 +CPU= 43.826
  02:15:20 STEP 1210 H=1210:00 +CPU= 30.152
  02:15:49 STEP 1211 H=1211:00 +CPU= 28.921
  02:16:39 STEP 1212 H=1212:00 +CPU= 48.513
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001212
  02:17:09 STEP 1213 H=1213:00 +CPU= 29.342
  02:17:39 STEP 1214 H=1214:00 +CPU= 30.208
  02:18:24 STEP 1215 H=1215:00 +CPU= 43.886
  02:18:55 STEP 1216 H=1216:00 +CPU= 30.136
  02:19:24 STEP 1217 H=1217:00 +CPU= 28.922
  02:20:13 STEP 1218 H=1218:00 +CPU= 45.569
  02:20:43 STEP 1219 H=1219:00 +CPU= 29.505
  02:21:13 STEP 1220 H=1220:00 +CPU= 30.362
  02:21:58 STEP 1221 H=1221:00 +CPU= 44.941
  02:22:31 STEP 1222 H=1222:00 +CPU= 30.318
  02:23:03 STEP 1223 H=1223:00 +CPU= 30.810
  02:24:01 STEP 1224 H=1224:00 +CPU= 50.883
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_50.zip
Uploading the intermediate file: upload_file_50.zip
  02:24:32 STEP 1225 H=1225:00 +CPU= 28.760
  02:24:58 STEP 1226 H=1226:00 +CPU= 26.163
  02:25:39 STEP 1227 H=1227:00 +CPU= 39.798
  02:26:10 STEP 1228 H=1228:00 +CPU= 30.513
  02:26:34 STEP 1229 H=1229:00 +CPU= 23.944
  02:27:14 STEP 1230 H=1230:00 +CPU= 39.703
  02:27:42 STEP 1231 H=1231:00 +CPU= 28.281
  02:28:12 STEP 1232 H=1232:00 +CPU= 29.550
  02:28:56 STEP 1233 H=1233:00 +CPU= 43.753
  02:29:22 STEP 1234 H=1234:00 +CPU= 25.818
  02:29:48 STEP 1235 H=1235:00 +CPU= 25.226
  02:30:33 STEP 1236 H=1236:00 +CPU= 44.724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001236
  02:31:00 STEP 1237 H=1237:00 +CPU= 27.520
  02:31:33 STEP 1238 H=1238:00 +CPU= 30.219
  02:32:17 STEP 1239 H=1239:00 +CPU= 44.355
  02:32:48 STEP 1240 H=1240:00 +CPU= 30.564
  02:33:14 STEP 1241 H=1241:00 +CPU= 26.056
  02:33:58 STEP 1242 H=1242:00 +CPU= 43.911
  02:34:27 STEP 1243 H=1243:00 +CPU= 28.241
  02:34:56 STEP 1244 H=1244:00 +CPU= 29.325
  02:35:35 STEP 1245 H=1245:00 +CPU= 38.088
  02:36:05 STEP 1246 H=1246:00 +CPU= 29.699
  02:36:32 STEP 1247 H=1247:00 +CPU= 27.220
  02:37:30 STEP 1248 H=1248:00 +CPU= 50.032
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894_51.zip
Uploading the intermediate file: upload_file_51.zip
  02:37:59 STEP 1249 H=1249:00 +CPU= 28.577
  02:38:29 STEP 1250 H=1250:00 +CPU= 29.992
  02:39:11 STEP 1251 H=1251:00 +CPU= 41.091
  02:39:40 STEP 1252 H=1252:00 +CPU= 28.981
  02:40:08 STEP 1253 H=1253:00 +CPU= 28.074
  02:40:43 STEP 1254 H=1254:00 +CPU= 34.303
  02:41:11 STEP 1255 H=1255:00 +CPU= 28.014
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0944
(argv4) batchid: 965
(argv5) wuid: 12182588
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182588
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_56ae93ffb4f1bd909ed29bf21d5376c3 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0944_1996050100_123_965_12182588.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0944_1996050100_123_965_12182588.zip
ic_ancil_file: ic_ancil_12182588
ifsdata_file: ifsdata_12182588
climate_data_file: clim_data_12182588
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6c963148c9bd770366a72b74baaefec4 to: /var/lib/boinc-client/slots/2/ic_ancil_12182588.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12182588.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f672791a5af978217ffcaa90d3d02f2a to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182588.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12182588.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182588.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12182588.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12182588.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12182588.xml
-- Model is restarting --
Adjusting last_iter, 1256, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12182588.xml
last_cpu_time: 43951
upload_file_number: 52
last_iter: 1247
last_upload: 4492800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0944_1996050100_123_965_12182588_0_r272712894
The child process has been launched with process id: 22943
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL04:1:1:22943:22943] [20221223:024228:1671759748:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:45:05 STEP 1248 H=1248:00 +CPU= 44.978
forrtl: severe (41): insufficient virtual memory
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001D2B9CB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000D78293  call_sl_heap_              92  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:48:58 1534452 22267698 oifs_43r3_ps_0944_1996050100_123_965_12182588_0 3,715,200 36,090 0.0097
09 May 2023 15:48:44 1534452 22267698 oifs_43r3_ps_0944_1996050100_123_965_12182588_0 2,592,000 24,786 0.0096
09 May 2023 15:48:30 1534452 22267698 oifs_43r3_ps_0944_1996050100_123_965_12182588_0 1,728,000 16,729 0.0097
09 May 2023 15:48:18 1534452 22267698 oifs_43r3_ps_0944_1996050100_123_965_12182588_0 864,000 8,593 0.0099


©2024 cpdn.org