climateprediction.net home page
Task 22438355

Task 22438355

Name oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0
Workunit 12286734
Created 7 Jun 2024, 17:56:24 UTC
Sent 7 Jun 2024, 21:22:26 UTC
Report deadline 6 Aug 2024, 21:22:26 UTC
Received 8 Jun 2024, 21:07:03 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1545820
Run time 22 hours 35 min 16 sec
CPU time 22 hours 35 min 16 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 6.25 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,023.04 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,365.41 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
older/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:test-System-Product-Name:1:1:8876:8876] [20240608:061630:1717845390:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:17:26 STEP  816 H= 204:00 +CPU= 48.382
  06:18:15 STEP  817 H= 204:15 +CPU= 48.839
  06:18:44 STEP  818 H= 204:30 +CPU= 28.147
  06:19:11 STEP  819 H= 204:45 +CPU= 26.983
  06:19:37 STEP  820 H= 205:00 +CPU= 26.771
  06:19:58 STEP  821 H= 205:15 +CPU= 20.895
  06:20:19 STEP  822 H= 205:30 +CPU= 20.399
  06:20:43 STEP  823 H= 205:45 +CPU= 24.533
  06:21:42 STEP  824 H= 206:00 +CPU= 58.826
  06:22:38 STEP  825 H= 206:15 +CPU= 54.504
  06:23:29 STEP  826 H= 206:30 +CPU= 50.846
  06:24:26 STEP  827 H= 206:45 +CPU= 55.911
  06:25:29 STEP  828 H= 207:00 +CPU= 63.377
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000828
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000828
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000828
  06:26:21 STEP  829 H= 207:15 +CPU= 51.803
  06:27:16 STEP  830 H= 207:30 +CPU= 54.724
  06:28:10 STEP  831 H= 207:45 +CPU= 52.682
  06:29:08 STEP  832 H= 208:00 +CPU= 57.765
  06:30:08 STEP  833 H= 208:15 +CPU= 60.033
  06:30:59 STEP  834 H= 208:30 +CPU= 49.977
  06:31:51 STEP  835 H= 208:45 +CPU= 52.192
  06:32:48 STEP  836 H= 209:00 +CPU= 56.626
  06:33:39 STEP  837 H= 209:15 +CPU= 50.388
  06:34:31 STEP  838 H= 209:30 +CPU= 52.079
  06:35:27 STEP  839 H= 209:45 +CPU= 55.158
  06:36:27 STEP  840 H= 210:00 +CPU= 60.024
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000840
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000840
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000840
  06:37:12 STEP  841 H= 210:15 +CPU= 44.459
  06:38:12 STEP  842 H= 210:30 +CPU= 59.591
  06:39:14 STEP  843 H= 210:45 +CPU= 61.061
  06:40:13 STEP  844 H= 211:00 +CPU= 59.326
  06:41:12 STEP  845 H= 211:15 +CPU= 57.892
  06:42:11 STEP  846 H= 211:30 +CPU= 58.719
  06:43:16 STEP  847 H= 211:45 +CPU= 63.863
  06:44:22 STEP  848 H= 212:00 +CPU= 65.371
  06:45:23 STEP  849 H= 212:15 +CPU= 61.033
  06:46:27 STEP  850 H= 212:30 +CPU= 63.468
  06:47:23 STEP  851 H= 212:45 +CPU= 55.166
  06:48:31 STEP  852 H= 213:00 +CPU= 67.929
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000852
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000852
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000852
  06:49:39 STEP  853 H= 213:15 +CPU= 67.657
  06:50:50 STEP  854 H= 213:30 +CPU= 69.719
  06:51:46 STEP  855 H= 213:45 +CPU= 55.894
  06:52:42 STEP  856 H= 214:00 +CPU= 56.152
  06:53:48 STEP  857 H= 214:15 +CPU= 64.866
  06:54:54 STEP  858 H= 214:30 +CPU= 65.538
  06:55:54 STEP  859 H= 214:45 +CPU= 60.462
  06:56:59 STEP  860 H= 215:00 +CPU= 64.826
  06:58:07 STEP  861 H= 215:15 +CPU= 66.535
  06:59:02 STEP  862 H= 215:30 +CPU= 54.022
  07:00:03 STEP  863 H= 215:45 +CPU= 61.124
  07:01:25 STEP  864 H= 216:00 +CPU= 80.988
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000864
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000864
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000852
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000852
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000852
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_8.zip
Uploading the intermediate file: upload_file_8.zip
  07:02:27 STEP  865 H= 216:15 +CPU= 60.022
  07:03:34 STEP  866 H= 216:30 +CPU= 66.237
  07:04:35 STEP  867 H= 216:45 +CPU= 59.982
  07:05:27 STEP  868 H= 217:00 +CPU= 51.389
  07:06:22 STEP  869 H= 217:15 +CPU= 54.295
  07:07:23 STEP  870 H= 217:30 +CPU= 60.882
  07:08:21 STEP  871 H= 217:45 +CPU= 56.820
  07:09:20 STEP  872 H= 218:00 +CPU= 58.868
  07:10:30 STEP  873 H= 218:15 +CPU= 69.908
  07:11:31 STEP  874 H= 218:30 +CPU= 60.289
  07:12:31 STEP  875 H= 218:45 +CPU= 58.766
  07:13:53 STEP  876 H= 219:00 +CPU= 81.964
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000876
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000876
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000876
  07:15:01 STEP  877 H= 219:15 +CPU= 66.735
  07:16:09 STEP  878 H= 219:30 +CPU= 67.016
  07:17:19 STEP  879 H= 219:45 +CPU= 70.096
  07:18:27 STEP  880 H= 220:00 +CPU= 67.875
  07:19:37 STEP  881 H= 220:15 +CPU= 69.202
  07:20:50 STEP  882 H= 220:30 +CPU= 72.061
  07:21:57 STEP  883 H= 220:45 +CPU= 66.901
  07:23:18 STEP  884 H= 221:00 +CPU= 79.756
  07:24:25 STEP  885 H= 221:15 +CPU= 65.892
  07:25:44 STEP  886 H= 221:30 +CPU= 78.574
  07:26:54 STEP  887 H= 221:45 +CPU= 69.603
  07:28:20 STEP  888 H= 222:00 +CPU= 84.561
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000888
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000888
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000888
  07:29:28 STEP  889 H= 222:15 +CPU= 67.407
  07:30:26 STEP  890 H= 222:30 +CPU= 57.968
  07:31:26 STEP  891 H= 222:45 +CPU= 59.868
  07:32:35 STEP  892 H= 223:00 +CPU= 67.598
  07:33:44 STEP  893 H= 223:15 +CPU= 68.767
  07:34:48 STEP  894 H= 223:30 +CPU= 64.070
  07:35:49 STEP  895 H= 223:45 +CPU= 60.179
  07:37:02 STEP  896 H= 224:00 +CPU= 71.813
  07:38:05 STEP  897 H= 224:15 +CPU= 62.790
  07:39:20 STEP  898 H= 224:30 +CPU= 74.437
  07:40:23 STEP  899 H= 224:45 +CPU= 63.061
  07:41:50 STEP  900 H= 225:00 +CPU= 86.326
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000900
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000900
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000900
  07:43:01 STEP  901 H= 225:15 +CPU= 70.003
  07:44:10 STEP  902 H= 225:30 +CPU= 68.750
  07:45:20 STEP  903 H= 225:45 +CPU= 69.049
  07:46:31 STEP  904 H= 226:00 +CPU= 70.859
  07:47:40 STEP  905 H= 226:15 +CPU= 68.526
  07:48:45 STEP  906 H= 226:30 +CPU= 64.499
  07:49:49 STEP  907 H= 226:45 +CPU= 62.752
  07:50:51 STEP  908 H= 227:00 +CPU= 62.274
  07:52:07 STEP  909 H= 227:15 +CPU= 75.494
  07:53:19 STEP  910 H= 227:30 +CPU= 71.175
  07:54:25 STEP  911 H= 227:45 +CPU= 65.512
  07:55:52 STEP  912 H= 228:00 +CPU= 85.845
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000912
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000912
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000912
  07:56:52 STEP  913 H= 228:15 +CPU= 59.824
  07:58:05 STEP  914 H= 228:30 +CPU= 71.539
  07:59:07 STEP  915 H= 228:45 +CPU= 62.051
  08:00:16 STEP  916 H= 229:00 +CPU= 67.735
  08:01:23 STEP  917 H= 229:15 +CPU= 66.582
  08:02:37 STEP  918 H= 229:30 +CPU= 72.954
  08:03:48 STEP  919 H= 229:45 +CPU= 71.481
  08:04:57 STEP  920 H= 230:00 +CPU= 66.617
  08:05:57 STEP  921 H= 230:15 +CPU= 60.103
  08:07:02 STEP  922 H= 230:30 +CPU= 64.733
  08:08:13 STEP  923 H= 230:45 +CPU= 69.906
  08:09:31 STEP  924 H= 231:00 +CPU= 78.376
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000924
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000924
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000924
  08:10:35 STEP  925 H= 231:15 +CPU= 62.686
  08:11:36 STEP  926 H= 231:30 +CPU= 61.754
  08:12:44 STEP  927 H= 231:45 +CPU= 65.575
  08:13:51 STEP  928 H= 232:00 +CPU= 66.559
  08:14:59 STEP  929 H= 232:15 +CPU= 67.252
  08:16:08 STEP  930 H= 232:30 +CPU= 68.598
  08:17:16 STEP  931 H= 232:45 +CPU= 66.892
  08:18:30 STEP  932 H= 233:00 +CPU= 73.579
  08:19:37 STEP  933 H= 233:15 +CPU= 67.268
  08:20:50 STEP  934 H= 233:30 +CPU= 71.875
  08:21:49 STEP  935 H= 233:45 +CPU= 58.663
  08:23:09 STEP  936 H= 234:00 +CPU= 79.329
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000936
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000936
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000936
  08:24:12 STEP  937 H= 234:15 +CPU= 62.323
  08:25:17 STEP  938 H= 234:30 +CPU= 65.046
  08:26:25 STEP  939 H= 234:45 +CPU= 67.122
  08:27:29 STEP  940 H= 235:00 +CPU= 63.463
  08:28:37 STEP  941 H= 235:15 +CPU= 68.175
  08:29:45 STEP  942 H= 235:30 +CPU= 67.225
  08:30:51 STEP  943 H= 235:45 +CPU= 65.489
  08:32:01 STEP  944 H= 236:00 +CPU= 68.936
  08:33:04 STEP  945 H= 236:15 +CPU= 62.652
  08:34:04 STEP  946 H= 236:30 +CPU= 59.663
  08:35:20 STEP  947 H= 236:45 +CPU= 75.011
  08:36:33 STEP  948 H= 237:00 +CPU= 72.673
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000948
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000948
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000948
  08:37:34 STEP  949 H= 237:15 +CPU= 59.856
  08:38:53 STEP  950 H= 237:30 +CPU= 78.325
  08:40:11 STEP  951 H= 237:45 +CPU= 77.372
  08:41:23 STEP  952 H= 238:00 +CPU= 70.802
  08:42:38 STEP  953 H= 238:15 +CPU= 73.803
  08:43:42 STEP  954 H= 238:30 +CPU= 63.784
  08:44:43 STEP  955 H= 238:45 +CPU= 60.189
  08:45:51 STEP  956 H= 239:00 +CPU= 67.369
  08:46:58 STEP  957 H= 239:15 +CPU= 67.184
  08:48:12 STEP  958 H= 239:30 +CPU= 72.862
  08:49:19 STEP  959 H= 239:45 +CPU= 67.443
  08:51:01 STEP  960 H= 240:00 +CPU=100.927
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000960
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000960
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000948
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000948
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000948
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_9.zip
Uploading the intermediate file: upload_file_9.zip
  08:52:10 STEP  961 H= 240:15 +CPU= 67.586
  08:53:17 STEP  962 H= 240:30 +CPU= 66.599
  08:54:19 STEP  963 H= 240:45 +CPU= 61.466
  08:55:31 STEP  964 H= 241:00 +CPU= 70.896
  08:56:38 STEP  965 H= 241:15 +CPU= 66.565
  08:57:48 STEP  966 H= 241:30 +CPU= 69.264
  08:58:59 STEP  967 H= 241:45 +CPU= 70.859
  09:00:09 STEP  968 H= 242:00 +CPU= 68.395
  09:01:14 STEP  969 H= 242:15 +CPU= 64.837
  09:02:25 STEP  970 H= 242:30 +CPU= 69.730
  09:03:28 STEP  971 H= 242:45 +CPU= 61.886
  09:04:45 STEP  972 H= 243:00 +CPU= 76.817
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000972
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000972
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000972
  09:05:45 STEP  973 H= 243:15 +CPU= 59.232
  09:06:59 STEP  974 H= 243:30 +CPU= 73.539
  09:08:07 STEP  975 H= 243:45 +CPU= 66.928
  09:09:17 STEP  976 H= 244:00 +CPU= 68.969
  09:10:27 STEP  977 H= 244:15 +CPU= 69.356
  09:11:40 STEP  978 H= 244:30 +CPU= 73.389
  09:12:51 STEP  979 H= 244:45 +CPU= 69.491
  09:14:05 STEP  980 H= 245:00 +CPU= 73.501
  09:15:22 STEP  981 H= 245:15 +CPU= 76.140
  09:16:39 STEP  982 H= 245:30 +CPU= 75.597
  09:17:52 STEP  983 H= 245:45 +CPU= 72.097
  09:19:09 STEP  984 H= 246:00 +CPU= 76.560
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000984
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000984
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000984
  09:20:19 STEP  985 H= 246:15 +CPU= 69.175
  09:21:29 STEP  986 H= 246:30 +CPU= 69.187
  09:22:37 STEP  987 H= 246:45 +CPU= 68.007
  09:23:50 STEP  988 H= 247:00 +CPU= 71.560
  09:24:59 STEP  989 H= 247:15 +CPU= 69.506
  09:26:11 STEP  990 H= 247:30 +CPU= 70.902
  09:27:21 STEP  991 H= 247:45 +CPU= 69.883
  09:28:27 STEP  992 H= 248:00 +CPU= 64.849
  09:29:45 STEP  993 H= 248:15 +CPU= 77.137
  09:30:51 STEP  994 H= 248:30 +CPU= 66.027
  09:32:00 STEP  995 H= 248:45 +CPU= 67.957
  09:33:24 STEP  996 H= 249:00 +CPU= 83.267
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+000996
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+000996
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+000996
  09:34:37 STEP  997 H= 249:15 +CPU= 72.552
  09:35:46 STEP  998 H= 249:30 +CPU= 68.173
  09:37:03 STEP  999 H= 249:45 +CPU= 76.244
  09:38:20 STEP 1000 H= 250:00 +CPU= 75.326
  09:39:36 STEP 1001 H= 250:15 +CPU= 75.654
  09:40:42 STEP 1002 H= 250:30 +CPU= 65.399
  09:41:51 STEP 1003 H= 250:45 +CPU= 69.071
  09:42:54 STEP 1004 H= 251:00 +CPU= 61.778
  09:44:02 STEP 1005 H= 251:15 +CPU= 67.283
  09:45:09 STEP 1006 H= 251:30 +CPU= 65.759
  09:46:16 STEP 1007 H= 251:45 +CPU= 66.897
  09:47:54 STEP 1008 H= 252:00 +CPU= 97.012
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001008
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001008
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001008
  09:49:13 STEP 1009 H= 252:15 +CPU= 78.276
  09:50:15 STEP 1010 H= 252:30 +CPU= 61.652
  09:51:23 STEP 1011 H= 252:45 +CPU= 67.500
  09:52:29 STEP 1012 H= 253:00 +CPU= 65.896
  09:53:40 STEP 1013 H= 253:15 +CPU= 70.367
  09:54:48 STEP 1014 H= 253:30 +CPU= 67.095
  09:56:04 STEP 1015 H= 253:45 +CPU= 75.058
  09:57:06 STEP 1016 H= 254:00 +CPU= 61.047
  09:58:19 STEP 1017 H= 254:15 +CPU= 71.915
  09:59:26 STEP 1018 H= 254:30 +CPU= 66.919
  10:00:31 STEP 1019 H= 254:45 +CPU= 64.245
  10:01:56 STEP 1020 H= 255:00 +CPU= 84.005
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001020
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001020
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001020
  10:03:01 STEP 1021 H= 255:15 +CPU= 64.594
  10:04:23 STEP 1022 H= 255:30 +CPU= 81.838
  10:05:38 STEP 1023 H= 255:45 +CPU= 74.350
  10:06:47 STEP 1024 H= 256:00 +CPU= 68.332
  10:07:59 STEP 1025 H= 256:15 +CPU= 71.517
  10:09:07 STEP 1026 H= 256:30 +CPU= 66.436
  10:10:18 STEP 1027 H= 256:45 +CPU= 70.234
  10:11:30 STEP 1028 H= 257:00 +CPU= 71.175
  10:12:49 STEP 1029 H= 257:15 +CPU= 78.904
  10:13:56 STEP 1030 H= 257:30 +CPU= 65.879
  10:15:02 STEP 1031 H= 257:45 +CPU= 65.924
  10:16:34 STEP 1032 H= 258:00 +CPU= 91.879
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001032
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001032
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001032
  10:17:42 STEP 1033 H= 258:15 +CPU= 66.559
  10:18:55 STEP 1034 H= 258:30 +CPU= 72.629
  10:20:07 STEP 1035 H= 258:45 +CPU= 71.826
  10:21:10 STEP 1036 H= 259:00 +CPU= 61.859
  10:22:24 STEP 1037 H= 259:15 +CPU= 73.349
  10:23:37 STEP 1038 H= 259:30 +CPU= 72.639
  10:24:44 STEP 1039 H= 259:45 +CPU= 66.084
  10:25:52 STEP 1040 H= 260:00 +CPU= 67.463
  10:27:11 STEP 1041 H= 260:15 +CPU= 77.788
  10:28:15 STEP 1042 H= 260:30 +CPU= 63.479
  10:29:25 STEP 1043 H= 260:45 +CPU= 69.358
  10:30:45 STEP 1044 H= 261:00 +CPU= 79.307
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001044
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001044
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001044
  10:31:52 STEP 1045 H= 261:15 +CPU= 66.185
  10:33:00 STEP 1046 H= 261:30 +CPU= 68.253
  10:34:10 STEP 1047 H= 261:45 +CPU= 69.138
  10:35:26 STEP 1048 H= 262:00 +CPU= 76.249
  10:36:33 STEP 1049 H= 262:15 +CPU= 65.669
  10:37:34 STEP 1050 H= 262:30 +CPU= 61.119
  10:38:44 STEP 1051 H= 262:45 +CPU= 68.392
  10:40:01 STEP 1052 H= 263:00 +CPU= 77.094
  10:41:05 STEP 1053 H= 263:15 +CPU= 63.490
  10:42:18 STEP 1054 H= 263:30 +CPU= 71.800
  10:43:27 STEP 1055 H= 263:45 +CPU= 68.904
  10:44:46 STEP 1056 H= 264:00 +CPU= 78.642
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001056
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001056
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001044
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001044
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001044
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_10.zip
Uploading the intermediate file: upload_file_10.zip
  10:45:50 STEP 1057 H= 264:15 +CPU= 62.777
  10:47:00 STEP 1058 H= 264:30 +CPU= 68.967
  10:48:10 STEP 1059 H= 264:45 +CPU= 68.441
  10:49:23 STEP 1060 H= 265:00 +CPU= 71.845
  10:50:38 STEP 1061 H= 265:15 +CPU= 74.814
  10:51:43 STEP 1062 H= 265:30 +CPU= 65.034
  10:52:40 STEP 1063 H= 265:45 +CPU= 56.826
  10:53:36 STEP 1064 H= 266:00 +CPU= 55.542
  10:54:48 STEP 1065 H= 266:15 +CPU= 72.088
  10:55:50 STEP 1066 H= 266:30 +CPU= 61.287
  10:57:02 STEP 1067 H= 266:45 +CPU= 71.228
  10:58:21 STEP 1068 H= 267:00 +CPU= 78.169
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001068
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001068
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001068
  10:59:26 STEP 1069 H= 267:15 +CPU= 64.688
  11:00:32 STEP 1070 H= 267:30 +CPU= 65.868
  11:01:30 STEP 1071 H= 267:45 +CPU= 57.447
  11:02:32 STEP 1072 H= 268:00 +CPU= 62.149
  11:03:35 STEP 1073 H= 268:15 +CPU= 62.336
  11:04:45 STEP 1074 H= 268:30 +CPU= 69.514
  11:05:41 STEP 1075 H= 268:45 +CPU= 56.472
  11:06:43 STEP 1076 H= 269:00 +CPU= 61.055
  11:07:46 STEP 1077 H= 269:15 +CPU= 63.251
  11:08:59 STEP 1078 H= 269:30 +CPU= 71.731
  11:09:58 STEP 1079 H= 269:45 +CPU= 59.295
  11:11:12 STEP 1080 H= 270:00 +CPU= 73.423
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001080
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001080
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001080
  11:12:12 STEP 1081 H= 270:15 +CPU= 59.529
  11:13:22 STEP 1082 H= 270:30 +CPU= 69.580
  11:14:28 STEP 1083 H= 270:45 +CPU= 65.761
  11:15:34 STEP 1084 H= 271:00 +CPU= 65.552
  11:16:43 STEP 1085 H= 271:15 +CPU= 69.493
  11:17:54 STEP 1086 H= 271:30 +CPU= 70.387
  11:18:51 STEP 1087 H= 271:45 +CPU= 57.005
  11:20:00 STEP 1088 H= 272:00 +CPU= 68.762
  11:21:06 STEP 1089 H= 272:15 +CPU= 65.151
  11:22:06 STEP 1090 H= 272:30 +CPU= 59.970
  11:23:14 STEP 1091 H= 272:45 +CPU= 67.210
  11:24:36 STEP 1092 H= 273:00 +CPU= 80.552
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001092
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001092
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001092
  11:25:59 STEP 1093 H= 273:15 +CPU= 83.481
  11:27:10 STEP 1094 H= 273:30 +CPU= 70.072
  11:28:21 STEP 1095 H= 273:45 +CPU= 70.389
  11:29:29 STEP 1096 H= 274:00 +CPU= 67.807
  11:30:33 STEP 1097 H= 274:15 +CPU= 63.858
  11:31:50 STEP 1098 H= 274:30 +CPU= 76.925
  11:32:57 STEP 1099 H= 274:45 +CPU= 66.209
  11:34:01 STEP 1100 H= 275:00 +CPU= 64.408
  11:35:12 STEP 1101 H= 275:15 +CPU= 69.892
  11:36:20 STEP 1102 H= 275:30 +CPU= 67.191
  11:37:27 STEP 1103 H= 275:45 +CPU= 66.306
  11:38:54 STEP 1104 H= 276:00 +CPU= 86.876
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001104
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001104
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001104
  11:40:05 STEP 1105 H= 276:15 +CPU= 69.918
  11:41:09 STEP 1106 H= 276:30 +CPU= 63.563
  11:42:17 STEP 1107 H= 276:45 +CPU= 67.225
  11:43:19 STEP 1108 H= 277:00 +CPU= 61.665
  11:44:25 STEP 1109 H= 277:15 +CPU= 65.484
  11:45:34 STEP 1110 H= 277:30 +CPU= 68.094
  11:46:36 STEP 1111 H= 277:45 +CPU= 61.791
  11:47:42 STEP 1112 H= 278:00 +CPU= 65.619
  11:48:48 STEP 1113 H= 278:15 +CPU= 66.427
  11:49:58 STEP 1114 H= 278:30 +CPU= 69.050
  11:51:06 STEP 1115 H= 278:45 +CPU= 68.183
  11:52:25 STEP 1116 H= 279:00 +CPU= 78.270
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001116
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001116
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001116
  11:53:35 STEP 1117 H= 279:15 +CPU= 69.189
  11:54:43 STEP 1118 H= 279:30 +CPU= 68.035
  11:55:37 STEP 1119 H= 279:45 +CPU= 54.423
  11:56:44 STEP 1120 H= 280:00 +CPU= 66.338
  11:57:49 STEP 1121 H= 280:15 +CPU= 64.047
  11:59:02 STEP 1122 H= 280:30 +CPU= 72.545
  12:00:07 STEP 1123 H= 280:45 +CPU= 64.962
  12:01:15 STEP 1124 H= 281:00 +CPU= 67.217
  12:02:28 STEP 1125 H= 281:15 +CPU= 72.932
  12:03:31 STEP 1126 H= 281:30 +CPU= 62.485
  12:04:45 STEP 1127 H= 281:45 +CPU= 73.106
  12:05:57 STEP 1128 H= 282:00 +CPU= 71.555
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001128
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001128
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001128
  12:07:09 STEP 1129 H= 282:15 +CPU= 70.972
  12:08:13 STEP 1130 H= 282:30 +CPU= 64.052
  12:09:18 STEP 1131 H= 282:45 +CPU= 63.760
  12:10:22 STEP 1132 H= 283:00 +CPU= 63.024
  12:11:25 STEP 1133 H= 283:15 +CPU= 62.798
  12:12:28 STEP 1134 H= 283:30 +CPU= 62.418
  12:13:27 STEP 1135 H= 283:45 +CPU= 58.960
  12:14:40 STEP 1136 H= 284:00 +CPU= 71.673
  12:15:47 STEP 1137 H= 284:15 +CPU= 65.681
  12:16:54 STEP 1138 H= 284:30 +CPU= 66.830
  12:17:56 STEP 1139 H= 284:45 +CPU= 62.038
  12:19:17 STEP 1140 H= 285:00 +CPU= 79.794
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001140
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001140
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001140
  12:20:25 STEP 1141 H= 285:15 +CPU= 67.876
  12:21:30 STEP 1142 H= 285:30 +CPU= 64.236
  12:22:35 STEP 1143 H= 285:45 +CPU= 64.637
  12:23:49 STEP 1144 H= 286:00 +CPU= 74.134
  12:24:54 STEP 1145 H= 286:15 +CPU= 64.445
  12:25:57 STEP 1146 H= 286:30 +CPU= 62.335
  12:27:05 STEP 1147 H= 286:45 +CPU= 67.304
  12:28:10 STEP 1148 H= 287:00 +CPU= 64.140
  12:29:19 STEP 1149 H= 287:15 +CPU= 68.055
  12:30:27 STEP 1150 H= 287:30 +CPU= 68.222
  12:31:44 STEP 1151 H= 287:45 +CPU= 76.424
  12:33:15 STEP 1152 H= 288:00 +CPU= 90.251
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001152
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001152
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001140
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001140
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001140
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_11.zip
Uploading the intermediate file: upload_file_11.zip
  12:34:30 STEP 1153 H= 288:15 +CPU= 73.926
  12:35:39 STEP 1154 H= 288:30 +CPU= 68.229
  12:36:29 STEP 1155 H= 288:45 +CPU= 49.428
  12:37:03 STEP 1156 H= 289:00 +CPU= 34.026
  12:37:37 STEP 1157 H= 289:15 +CPU= 33.385
  12:38:14 STEP 1158 H= 289:30 +CPU= 36.916
  12:38:53 STEP 1159 H= 289:45 +CPU= 38.639
  12:39:30 STEP 1160 H= 290:00 +CPU= 37.155
  12:40:10 STEP 1161 H= 290:15 +CPU= 39.953
  12:40:49 STEP 1162 H= 290:30 +CPU= 38.197
  12:41:25 STEP 1163 H= 290:45 +CPU= 36.355
  12:42:09 STEP 1164 H= 291:00 +CPU= 43.494
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001164
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001164
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001164
  12:42:44 STEP 1165 H= 291:15 +CPU= 34.063
  12:43:19 STEP 1166 H= 291:30 +CPU= 35.311
  12:43:55 STEP 1167 H= 291:45 +CPU= 35.744
  12:44:31 STEP 1168 H= 292:00 +CPU= 35.603
  12:45:06 STEP 1169 H= 292:15 +CPU= 35.685
  12:45:42 STEP 1170 H= 292:30 +CPU= 35.398
  12:46:17 STEP 1171 H= 292:45 +CPU= 35.089
  12:46:55 STEP 1172 H= 293:00 +CPU= 38.084
  12:47:31 STEP 1173 H= 293:15 +CPU= 35.381
  12:48:09 STEP 1174 H= 293:30 +CPU= 38.367
  12:48:46 STEP 1175 H= 293:45 +CPU= 37.086
  12:49:31 STEP 1176 H= 294:00 +CPU= 44.919
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001176
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001176
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001176
  12:50:08 STEP 1177 H= 294:15 +CPU= 36.512
  12:50:45 STEP 1178 H= 294:30 +CPU= 36.486
  12:51:19 STEP 1179 H= 294:45 +CPU= 33.776
  12:51:55 STEP 1180 H= 295:00 +CPU= 36.070
  12:52:34 STEP 1181 H= 295:15 +CPU= 38.120
  12:53:11 STEP 1182 H= 295:30 +CPU= 36.763
  12:53:47 STEP 1183 H= 295:45 +CPU= 36.096
  12:54:26 STEP 1184 H= 296:00 +CPU= 38.172
  12:55:03 STEP 1185 H= 296:15 +CPU= 37.257
  12:55:40 STEP 1186 H= 296:30 +CPU= 36.846
  12:56:19 STEP 1187 H= 296:45 +CPU= 38.226
  12:57:07 STEP 1188 H= 297:00 +CPU= 47.758
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001188
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001188
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001188
  12:57:42 STEP 1189 H= 297:15 +CPU= 35.350
  12:58:19 STEP 1190 H= 297:30 +CPU= 35.983
  12:58:54 STEP 1191 H= 297:45 +CPU= 34.499
  12:59:33 STEP 1192 H= 298:00 +CPU= 38.993
  13:00:11 STEP 1193 H= 298:15 +CPU= 38.168
  13:00:49 STEP 1194 H= 298:30 +CPU= 37.407
  13:01:28 STEP 1195 H= 298:45 +CPU= 38.848
  13:02:02 STEP 1196 H= 299:00 +CPU= 33.725
  13:02:38 STEP 1197 H= 299:15 +CPU= 35.195
  13:03:11 STEP 1198 H= 299:30 +CPU= 33.554
  13:03:45 STEP 1199 H= 299:45 +CPU= 33.607
  13:04:36 STEP 1200 H= 300:00 +CPU= 50.238
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001200
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001200
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001200
  13:05:17 STEP 1201 H= 300:15 +CPU= 40.555
  13:05:54 STEP 1202 H= 300:30 +CPU= 37.527
  13:06:28 STEP 1203 H= 300:45 +CPU= 33.374
  13:07:04 STEP 1204 H= 301:00 +CPU= 35.599
  13:07:42 STEP 1205 H= 301:15 +CPU= 37.510
  13:08:18 STEP 1206 H= 301:30 +CPU= 35.091
  13:08:56 STEP 1207 H= 301:45 +CPU= 37.315
  13:09:36 STEP 1208 H= 302:00 +CPU= 40.329
  13:10:13 STEP 1209 H= 302:15 +CPU= 37.058
  13:10:47 STEP 1210 H= 302:30 +CPU= 33.843
  13:11:21 STEP 1211 H= 302:45 +CPU= 33.149
  13:12:03 STEP 1212 H= 303:00 +CPU= 41.958
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001212
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001212
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001212
  13:12:39 STEP 1213 H= 303:15 +CPU= 35.893
  13:13:15 STEP 1214 H= 303:30 +CPU= 35.922
  13:13:55 STEP 1215 H= 303:45 +CPU= 38.687
  13:14:31 STEP 1216 H= 304:00 +CPU= 36.262
  13:15:05 STEP 1217 H= 304:15 +CPU= 33.647
  13:15:39 STEP 1218 H= 304:30 +CPU= 33.670
  13:16:13 STEP 1219 H= 304:45 +CPU= 34.180
  13:16:52 STEP 1220 H= 305:00 +CPU= 38.600
  13:17:28 STEP 1221 H= 305:15 +CPU= 35.821
  13:18:05 STEP 1222 H= 305:30 +CPU= 36.365
  13:18:42 STEP 1223 H= 305:45 +CPU= 37.103
  13:19:23 STEP 1224 H= 306:00 +CPU= 40.704
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001224
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001224
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001224
  13:19:56 STEP 1225 H= 306:15 +CPU= 33.307
  13:20:30 STEP 1226 H= 306:30 +CPU= 33.448
  13:21:05 STEP 1227 H= 306:45 +CPU= 34.926
  13:21:42 STEP 1228 H= 307:00 +CPU= 36.261
  13:22:18 STEP 1229 H= 307:15 +CPU= 36.465
  13:22:58 STEP 1230 H= 307:30 +CPU= 39.691
  13:23:34 STEP 1231 H= 307:45 +CPU= 34.841
  13:24:09 STEP 1232 H= 308:00 +CPU= 34.842
  13:24:46 STEP 1233 H= 308:15 +CPU= 36.307
  13:25:21 STEP 1234 H= 308:30 +CPU= 34.894
  13:25:54 STEP 1235 H= 308:45 +CPU= 32.786
  13:26:36 STEP 1236 H= 309:00 +CPU= 41.890
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001236
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001236
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001236
  13:27:17 STEP 1237 H= 309:15 +CPU= 40.149
  13:27:50 STEP 1238 H= 309:30 +CPU= 33.253
  13:28:26 STEP 1239 H= 309:45 +CPU= 34.994
  13:28:58 STEP 1240 H= 310:00 +CPU= 32.276
  13:29:29 STEP 1241 H= 310:15 +CPU= 31.340
  13:30:05 STEP 1242 H= 310:30 +CPU= 35.924
  13:30:40 STEP 1243 H= 310:45 +CPU= 34.893
  13:31:19 STEP 1244 H= 311:00 +CPU= 37.994
  13:31:54 STEP 1245 H= 311:15 +CPU= 35.319
  13:32:33 STEP 1246 H= 311:30 +CPU= 38.369
  13:33:09 STEP 1247 H= 311:45 +CPU= 35.899
  13:33:58 STEP 1248 H= 312:00 +CPU= 48.431
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001248
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001248
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001236
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001236
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001236
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_12.zip
Uploading the intermediate file: upload_file_12.zip
  13:34:29 STEP 1249 H= 312:15 +CPU= 31.344
  13:35:04 STEP 1250 H= 312:30 +CPU= 33.981
  13:35:36 STEP 1251 H= 312:45 +CPU= 31.909
  13:36:15 STEP 1252 H= 313:00 +CPU= 38.702
  13:36:49 STEP 1253 H= 313:15 +CPU= 34.139
  13:37:27 STEP 1254 H= 313:30 +CPU= 37.916
  13:38:01 STEP 1255 H= 313:45 +CPU= 33.667
  13:38:35 STEP 1256 H= 314:00 +CPU= 33.239
  13:39:07 STEP 1257 H= 314:15 +CPU= 32.453
  13:39:44 STEP 1258 H= 314:30 +CPU= 35.588
  13:40:17 STEP 1259 H= 314:45 +CPU= 33.478
  13:41:00 STEP 1260 H= 315:00 +CPU= 42.761
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001260
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001260
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001260
  13:41:37 STEP 1261 H= 315:15 +CPU= 36.638
  13:42:17 STEP 1262 H= 315:30 +CPU= 39.453
  13:42:55 STEP 1263 H= 315:45 +CPU= 37.180
  13:43:34 STEP 1264 H= 316:00 +CPU= 38.764
  13:44:12 STEP 1265 H= 316:15 +CPU= 38.544
  13:44:48 STEP 1266 H= 316:30 +CPU= 35.523
  13:45:25 STEP 1267 H= 316:45 +CPU= 36.473
  13:46:01 STEP 1268 H= 317:00 +CPU= 36.305
  13:46:38 STEP 1269 H= 317:15 +CPU= 35.903
  13:47:12 STEP 1270 H= 317:30 +CPU= 34.358
  13:47:48 STEP 1271 H= 317:45 +CPU= 35.339
  13:48:32 STEP 1272 H= 318:00 +CPU= 44.073
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001272
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001272
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001272
  13:49:07 STEP 1273 H= 318:15 +CPU= 35.234
  13:49:47 STEP 1274 H= 318:30 +CPU= 39.264
  13:50:27 STEP 1275 H= 318:45 +CPU= 39.805
  13:51:06 STEP 1276 H= 319:00 +CPU= 38.989
  13:51:41 STEP 1277 H= 319:15 +CPU= 34.575
  13:52:17 STEP 1278 H= 319:30 +CPU= 35.883
  13:52:51 STEP 1279 H= 319:45 +CPU= 34.072
  13:53:27 STEP 1280 H= 320:00 +CPU= 36.002
  13:54:04 STEP 1281 H= 320:15 +CPU= 36.451
  13:54:39 STEP 1282 H= 320:30 +CPU= 35.614
  13:55:16 STEP 1283 H= 320:45 +CPU= 36.569
  13:56:01 STEP 1284 H= 321:00 +CPU= 44.537
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001284
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001284
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001284
  13:56:35 STEP 1285 H= 321:15 +CPU= 34.022
  13:57:11 STEP 1286 H= 321:30 +CPU= 35.769
  13:57:48 STEP 1287 H= 321:45 +CPU= 36.993
  13:58:23 STEP 1288 H= 322:00 +CPU= 34.171
  13:59:03 STEP 1289 H= 322:15 +CPU= 39.784
  13:59:34 STEP 1290 H= 322:30 +CPU= 30.675
  14:00:06 STEP 1291 H= 322:45 +CPU= 32.415
  14:00:43 STEP 1292 H= 323:00 +CPU= 36.359
  14:01:22 STEP 1293 H= 323:15 +CPU= 38.146
  14:01:56 STEP 1294 H= 323:30 +CPU= 33.804
  14:02:33 STEP 1295 H= 323:45 +CPU= 37.309
  14:03:24 STEP 1296 H= 324:00 +CPU= 49.957
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001296
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001296
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001296
  14:03:58 STEP 1297 H= 324:15 +CPU= 34.258
  14:04:29 STEP 1298 H= 324:30 +CPU= 30.424
  14:04:56 STEP 1299 H= 324:45 +CPU= 26.945
  14:05:26 STEP 1300 H= 325:00 +CPU= 30.206
  14:05:59 STEP 1301 H= 325:15 +CPU= 33.284
  14:06:32 STEP 1302 H= 325:30 +CPU= 32.498
  14:07:00 STEP 1303 H= 325:45 +CPU= 28.589
  14:07:28 STEP 1304 H= 326:00 +CPU= 27.470
  14:07:56 STEP 1305 H= 326:15 +CPU= 27.669
  14:08:27 STEP 1306 H= 326:30 +CPU= 30.826
  14:09:02 STEP 1307 H= 326:45 +CPU= 35.109
  14:09:43 STEP 1308 H= 327:00 +CPU= 40.557
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001308
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001308
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001308
  14:10:17 STEP 1309 H= 327:15 +CPU= 34.262
  14:10:54 STEP 1310 H= 327:30 +CPU= 36.897
  14:11:32 STEP 1311 H= 327:45 +CPU= 38.042
  14:12:09 STEP 1312 H= 328:00 +CPU= 36.309
  14:12:42 STEP 1313 H= 328:15 +CPU= 33.471
  14:13:16 STEP 1314 H= 328:30 +CPU= 33.486
  14:13:52 STEP 1315 H= 328:45 +CPU= 36.044
  14:14:27 STEP 1316 H= 329:00 +CPU= 33.958
  14:15:00 STEP 1317 H= 329:15 +CPU= 33.499
  14:15:35 STEP 1318 H= 329:30 +CPU= 34.202
  14:16:08 STEP 1319 H= 329:45 +CPU= 33.069
  14:16:51 STEP 1320 H= 330:00 +CPU= 42.598
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001320
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001320
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001320
  14:17:28 STEP 1321 H= 330:15 +CPU= 36.885
  14:18:03 STEP 1322 H= 330:30 +CPU= 35.017
  14:18:37 STEP 1323 H= 330:45 +CPU= 33.643
  14:19:14 STEP 1324 H= 331:00 +CPU= 36.785
  14:19:48 STEP 1325 H= 331:15 +CPU= 33.873
  14:20:22 STEP 1326 H= 331:30 +CPU= 33.409
  14:20:56 STEP 1327 H= 331:45 +CPU= 33.817
  14:21:32 STEP 1328 H= 332:00 +CPU= 36.125
  14:22:08 STEP 1329 H= 332:15 +CPU= 35.789
  14:22:41 STEP 1330 H= 332:30 +CPU= 32.794
  14:23:15 STEP 1331 H= 332:45 +CPU= 34.030
  14:23:58 STEP 1332 H= 333:00 +CPU= 42.371
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001332
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001332
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001332
  14:24:34 STEP 1333 H= 333:15 +CPU= 36.143
  14:25:09 STEP 1334 H= 333:30 +CPU= 34.557
  14:25:44 STEP 1335 H= 333:45 +CPU= 34.130
  14:26:19 STEP 1336 H= 334:00 +CPU= 34.518
  14:26:51 STEP 1337 H= 334:15 +CPU= 31.787
  14:27:25 STEP 1338 H= 334:30 +CPU= 33.597
  14:27:58 STEP 1339 H= 334:45 +CPU= 33.554
  14:28:36 STEP 1340 H= 335:00 +CPU= 37.236
  14:29:13 STEP 1341 H= 335:15 +CPU= 36.786
  14:29:47 STEP 1342 H= 335:30 +CPU= 33.947
  14:30:24 STEP 1343 H= 335:45 +CPU= 37.264
  14:31:12 STEP 1344 H= 336:00 +CPU= 46.963
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001344
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001344
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001332
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001332
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001332
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_13.zip
Uploading the intermediate file: upload_file_13.zip
  14:31:44 STEP 1345 H= 336:15 +CPU= 31.806
  14:32:17 STEP 1346 H= 336:30 +CPU= 32.682
  14:32:55 STEP 1347 H= 336:45 +CPU= 37.938
  14:33:30 STEP 1348 H= 337:00 +CPU= 35.008
  14:34:06 STEP 1349 H= 337:15 +CPU= 35.391
  14:34:43 STEP 1350 H= 337:30 +CPU= 37.109
  14:35:16 STEP 1351 H= 337:45 +CPU= 32.112
  14:35:49 STEP 1352 H= 338:00 +CPU= 33.385
  14:36:23 STEP 1353 H= 338:15 +CPU= 33.680
  14:36:59 STEP 1354 H= 338:30 +CPU= 35.847
  14:37:34 STEP 1355 H= 338:45 +CPU= 34.530
  14:38:18 STEP 1356 H= 339:00 +CPU= 44.247
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001356
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001356
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001356
  14:38:57 STEP 1357 H= 339:15 +CPU= 37.584
  14:39:30 STEP 1358 H= 339:30 +CPU= 32.661
  14:40:01 STEP 1359 H= 339:45 +CPU= 31.644
  14:40:37 STEP 1360 H= 340:00 +CPU= 35.119
  14:41:16 STEP 1361 H= 340:15 +CPU= 39.057
  14:41:53 STEP 1362 H= 340:30 +CPU= 37.045
  14:42:30 STEP 1363 H= 340:45 +CPU= 35.866
  14:43:07 STEP 1364 H= 341:00 +CPU= 36.765
  14:43:42 STEP 1365 H= 341:15 +CPU= 34.747
  14:44:16 STEP 1366 H= 341:30 +CPU= 33.635
  14:44:50 STEP 1367 H= 341:45 +CPU= 34.263
  14:45:35 STEP 1368 H= 342:00 +CPU= 44.269
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001368
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001368
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001368
  14:46:09 STEP 1369 H= 342:15 +CPU= 33.636
  14:46:45 STEP 1370 H= 342:30 +CPU= 36.222
  14:47:23 STEP 1371 H= 342:45 +CPU= 37.781
  14:47:48 STEP 1372 H= 343:00 +CPU= 24.731
  14:48:12 STEP 1373 H= 343:15 +CPU= 24.666
  14:48:37 STEP 1374 H= 343:30 +CPU= 24.205
  14:49:01 STEP 1375 H= 343:45 +CPU= 24.584
  14:49:26 STEP 1376 H= 344:00 +CPU= 24.833
  14:49:51 STEP 1377 H= 344:15 +CPU= 24.533
  14:50:15 STEP 1378 H= 344:30 +CPU= 23.584
  14:50:40 STEP 1379 H= 344:45 +CPU= 24.979
  14:51:11 STEP 1380 H= 345:00 +CPU= 31.110
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001380
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001380
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001380
  14:51:35 STEP 1381 H= 345:15 +CPU= 24.149
  14:52:00 STEP 1382 H= 345:30 +CPU= 24.453
  14:52:24 STEP 1383 H= 345:45 +CPU= 24.598
  14:52:50 STEP 1384 H= 346:00 +CPU= 25.256
  14:53:13 STEP 1385 H= 346:15 +CPU= 22.944
  14:53:38 STEP 1386 H= 346:30 +CPU= 25.112
  14:54:04 STEP 1387 H= 346:45 +CPU= 25.170
  14:54:29 STEP 1388 H= 347:00 +CPU= 25.705
  14:54:54 STEP 1389 H= 347:15 +CPU= 25.031
  14:55:19 STEP 1390 H= 347:30 +CPU= 24.544
  14:55:43 STEP 1391 H= 347:45 +CPU= 24.152
  14:56:20 STEP 1392 H= 348:00 +CPU= 36.280
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001392
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001392
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001392
  14:56:46 STEP 1393 H= 348:15 +CPU= 26.656
  14:57:11 STEP 1394 H= 348:30 +CPU= 24.846
  14:57:37 STEP 1395 H= 348:45 +CPU= 25.972
  14:58:05 STEP 1396 H= 349:00 +CPU= 27.251
  14:58:30 STEP 1397 H= 349:15 +CPU= 25.226
  14:58:56 STEP 1398 H= 349:30 +CPU= 25.359
  14:59:20 STEP 1399 H= 349:45 +CPU= 24.507
  14:59:46 STEP 1400 H= 350:00 +CPU= 25.400
  15:00:10 STEP 1401 H= 350:15 +CPU= 23.815
  15:00:35 STEP 1402 H= 350:30 +CPU= 25.190
  15:01:01 STEP 1403 H= 350:45 +CPU= 25.009
  15:01:31 STEP 1404 H= 351:00 +CPU= 30.279
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001404
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001404
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001404
  15:01:57 STEP 1405 H= 351:15 +CPU= 25.206
  15:02:22 STEP 1406 H= 351:30 +CPU= 25.447
  15:02:47 STEP 1407 H= 351:45 +CPU= 24.436
  15:03:12 STEP 1408 H= 352:00 +CPU= 25.401
  15:03:35 STEP 1409 H= 352:15 +CPU= 22.773
  15:04:00 STEP 1410 H= 352:30 +CPU= 24.855
  15:04:26 STEP 1411 H= 352:45 +CPU= 26.235
  15:04:55 STEP 1412 H= 353:00 +CPU= 28.168
  15:05:21 STEP 1413 H= 353:15 +CPU= 26.045
  15:05:46 STEP 1414 H= 353:30 +CPU= 25.166
  15:06:11 STEP 1415 H= 353:45 +CPU= 24.644
  15:06:42 STEP 1416 H= 354:00 +CPU= 30.768
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001416
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001416
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001416
  15:07:09 STEP 1417 H= 354:15 +CPU= 26.564
  15:07:35 STEP 1418 H= 354:30 +CPU= 25.984
  15:08:01 STEP 1419 H= 354:45 +CPU= 26.764
  15:08:27 STEP 1420 H= 355:00 +CPU= 24.887
  15:08:52 STEP 1421 H= 355:15 +CPU= 24.798
  15:09:17 STEP 1422 H= 355:30 +CPU= 25.569
  15:09:43 STEP 1423 H= 355:45 +CPU= 25.685
  15:10:08 STEP 1424 H= 356:00 +CPU= 24.942
  15:10:32 STEP 1425 H= 356:15 +CPU= 23.585
  15:10:56 STEP 1426 H= 356:30 +CPU= 24.338
  15:11:23 STEP 1427 H= 356:45 +CPU= 26.229
  15:11:55 STEP 1428 H= 357:00 +CPU= 31.586
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001428
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001428
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001428
  15:12:21 STEP 1429 H= 357:15 +CPU= 25.777
  15:12:46 STEP 1430 H= 357:30 +CPU= 25.415
  15:13:10 STEP 1431 H= 357:45 +CPU= 23.793
  15:13:36 STEP 1432 H= 358:00 +CPU= 25.873
  15:14:00 STEP 1433 H= 358:15 +CPU= 23.818
  15:14:24 STEP 1434 H= 358:30 +CPU= 23.870
  15:14:49 STEP 1435 H= 358:45 +CPU= 24.716
  15:15:16 STEP 1436 H= 359:00 +CPU= 26.574
  15:15:42 STEP 1437 H= 359:15 +CPU= 25.223
  15:16:06 STEP 1438 H= 359:30 +CPU= 24.501
  15:16:31 STEP 1439 H= 359:45 +CPU= 24.523
  15:17:03 STEP 1440 H= 360:00 +CPU= 32.250
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.3539793861E-21      I       0.1174221600E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3914414944E-02      I
I  Relat. Hum. I      0.4323486049E+00      I       0.4588117017E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.7633776004E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    322425748 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS806064.4 Mbytes/s, TOTAL TIME=    0.0096( 4.2%)
 15:17:03 STEP 1440 H= 360:00 +CPU= 32.250
 END CNT3
 NSTEP =  1440 CNT0   000000000  23.396  23.396  23.504   0.995               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        18590/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    328    0       46      1044068    18590 ************    64.6     2.4    62.2     4.4   124.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.98      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  677   10518.8   10518.8     21.96      0.00
   9 SPCM     - SPECTRAL COMP.                     624    2697.2    2697.2      5.19      0.00
  10 SCAN2M - PHYSICS                              625   21515.7   21515.7     41.46      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 52    1917.3    1917.3      0.31      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 27     139.1     139.1      0.01      0.00
  17 GRIDFPOS IN CNT4                               52      62.5      62.5      0.01      0.00
  27 RESTART FILES - WRITING                        13    7669.0    7669.0      0.31      0.00
  28 RESTART FILES - READING                         1    6419.5    6419.5      0.02      0.00
  29 SU4FPOS IN CNT4                                52       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                52    9113.8    9113.8      1.46      0.00
  31 POSDDH IN STEPO                                 3      26.0      26.0      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    625    9273.3    9273.3     17.87      0.00
  39 SU0YOMB                                         1     754.3     754.3      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    625      61.2      61.2      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              624    1420.8    1420.8      2.73      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              624     143.8     143.8      0.28      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 27       1.1       1.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1356     663.7     663.7      2.77      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      1300     359.6     359.6      1.44      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       1300     163.1     163.1      0.65      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1356     237.5     237.5      0.99      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1     188.0     188.0      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1356     160.1     160.1      0.67      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           1300     101.1     101.1      0.41      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1356     252.1     252.1      1.05      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           1300     126.5     126.5      0.51      0.00
 400 GSTATS                                      73253       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 68103       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    32433.6 CPU TIME   32179.2 VECTOR TIME    32179.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       40.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/test/slots/16/ICMGGh7zg+001440
Moving to projects directory: /home/test/slots/16/ICMSHh7zg+001440
Moving to projects directory: /home/test/slots/16/ICMUAh7zg+001440
Adding to the zip: /home/test/slots/16/NODE.001_01
Adding to the zip: /home/test/slots/16/ifs.stat
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMGGh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMUAh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12286734/ICMSHh7zg+001344
Zipping up the final file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
15:20:33 (8873): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0_r1115961342_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 21:07:05 1545820 22438355 oifs_43r3_bl_a3sb_2016092300_20_1017_12286734_0 1,295,100 81,316 0.0628


©2024 cpdn.org