climateprediction.net home page
Task 22433702

Task 22433702

Name oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0
Workunit 12282135
Created 7 Jun 2024, 11:17:36 UTC
Sent 7 Jun 2024, 11:22:35 UTC
Report deadline 6 Aug 2024, 11:22:35 UTC
Received 8 Jun 2024, 10:08:46 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1545820
Run time 22 hours 10 min 16 sec
CPU time 22 hours 10 min 16 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 6.25 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.93 MB
Peak swap size 5,980.81 MB
Peak disk usage 1,361.80 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
TEP  772 H= 193:00 +CPU= 48.626
  17:35:56 STEP  773 H= 193:15 +CPU= 52.118
  17:36:48 STEP  774 H= 193:30 +CPU= 51.402
  17:37:39 STEP  775 H= 193:45 +CPU= 51.193
  17:38:26 STEP  776 H= 194:00 +CPU= 46.164
  17:39:13 STEP  777 H= 194:15 +CPU= 46.466
  17:40:03 STEP  778 H= 194:30 +CPU= 50.255
  17:40:51 STEP  779 H= 194:45 +CPU= 47.381
  17:41:45 STEP  780 H= 195:00 +CPU= 53.135
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000780
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000780
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000780
  17:42:39 STEP  781 H= 195:15 +CPU= 54.180
  17:43:31 STEP  782 H= 195:30 +CPU= 51.424
  17:44:19 STEP  783 H= 195:45 +CPU= 47.747
  17:45:08 STEP  784 H= 196:00 +CPU= 48.168
  17:46:01 STEP  785 H= 196:15 +CPU= 53.240
  17:46:59 STEP  786 H= 196:30 +CPU= 57.354
  17:47:39 STEP  787 H= 196:45 +CPU= 39.854
  17:48:26 STEP  788 H= 197:00 +CPU= 46.867
  17:49:13 STEP  789 H= 197:15 +CPU= 45.948
  17:49:57 STEP  790 H= 197:30 +CPU= 43.376
  17:50:47 STEP  791 H= 197:45 +CPU= 49.486
  17:51:52 STEP  792 H= 198:00 +CPU= 63.210
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000792
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000792
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000792
  17:52:40 STEP  793 H= 198:15 +CPU= 48.028
  17:53:24 STEP  794 H= 198:30 +CPU= 42.742
  17:54:14 STEP  795 H= 198:45 +CPU= 49.944
  17:54:59 STEP  796 H= 199:00 +CPU= 44.573
  17:55:52 STEP  797 H= 199:15 +CPU= 53.164
  17:56:45 STEP  798 H= 199:30 +CPU= 52.790
  17:57:42 STEP  799 H= 199:45 +CPU= 56.423
  17:58:38 STEP  800 H= 200:00 +CPU= 55.133
  17:59:33 STEP  801 H= 200:15 +CPU= 53.747
  18:00:28 STEP  802 H= 200:30 +CPU= 54.500
  18:01:24 STEP  803 H= 200:45 +CPU= 55.374
  18:02:19 STEP  804 H= 201:00 +CPU= 55.064
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000804
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000804
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000804
  18:03:15 STEP  805 H= 201:15 +CPU= 55.494
  18:04:03 STEP  806 H= 201:30 +CPU= 48.231
  18:04:54 STEP  807 H= 201:45 +CPU= 50.885
  18:05:53 STEP  808 H= 202:00 +CPU= 59.042
  18:06:50 STEP  809 H= 202:15 +CPU= 56.366
  18:07:43 STEP  810 H= 202:30 +CPU= 52.065
  18:08:35 STEP  811 H= 202:45 +CPU= 50.887
  18:09:31 STEP  812 H= 203:00 +CPU= 55.475
  18:10:25 STEP  813 H= 203:15 +CPU= 54.534
  18:11:15 STEP  814 H= 203:30 +CPU= 48.931
  18:12:08 STEP  815 H= 203:45 +CPU= 52.048
  18:13:21 STEP  816 H= 204:00 +CPU= 72.905
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000816
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000816
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000816
  18:14:15 STEP  817 H= 204:15 +CPU= 53.835
  18:15:16 STEP  818 H= 204:30 +CPU= 58.988
  18:16:19 STEP  819 H= 204:45 +CPU= 62.035
  18:17:15 STEP  820 H= 205:00 +CPU= 55.063
  18:18:14 STEP  821 H= 205:15 +CPU= 59.288
  18:19:10 STEP  822 H= 205:30 +CPU= 54.984
  18:19:58 STEP  823 H= 205:45 +CPU= 47.462
  18:20:50 STEP  824 H= 206:00 +CPU= 51.470
  18:21:42 STEP  825 H= 206:15 +CPU= 52.201
  18:22:36 STEP  826 H= 206:30 +CPU= 53.320
  18:23:34 STEP  827 H= 206:45 +CPU= 57.266
  18:24:37 STEP  828 H= 207:00 +CPU= 62.834
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000828
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000828
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000828
  18:25:34 STEP  829 H= 207:15 +CPU= 57.052
  18:26:32 STEP  830 H= 207:30 +CPU= 57.692
  18:27:33 STEP  831 H= 207:45 +CPU= 59.936
  18:28:31 STEP  832 H= 208:00 +CPU= 58.074
  18:29:20 STEP  833 H= 208:15 +CPU= 48.695
  18:30:13 STEP  834 H= 208:30 +CPU= 52.501
  18:31:09 STEP  835 H= 208:45 +CPU= 55.511
  18:32:03 STEP  836 H= 209:00 +CPU= 53.748
  18:32:52 STEP  837 H= 209:15 +CPU= 48.805
  18:33:40 STEP  838 H= 209:30 +CPU= 48.002
  18:34:36 STEP  839 H= 209:45 +CPU= 55.619
  18:35:46 STEP  840 H= 210:00 +CPU= 69.120
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000840
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000840
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000840
  18:36:43 STEP  841 H= 210:15 +CPU= 56.187
  18:37:33 STEP  842 H= 210:30 +CPU= 49.832
  18:38:22 STEP  843 H= 210:45 +CPU= 48.780
  18:39:15 STEP  844 H= 211:00 +CPU= 52.246
  18:40:10 STEP  845 H= 211:15 +CPU= 55.140
  18:41:03 STEP  846 H= 211:30 +CPU= 52.368
  18:42:01 STEP  847 H= 211:45 +CPU= 56.931
  18:43:00 STEP  848 H= 212:00 +CPU= 58.631
  18:43:49 STEP  849 H= 212:15 +CPU= 48.458
  18:44:43 STEP  850 H= 212:30 +CPU= 53.278
  18:45:38 STEP  851 H= 212:45 +CPU= 55.169
  18:46:42 STEP  852 H= 213:00 +CPU= 62.735
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000852
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000852
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000852
  18:47:43 STEP  853 H= 213:15 +CPU= 60.419
  18:48:38 STEP  854 H= 213:30 +CPU= 55.127
  18:49:29 STEP  855 H= 213:45 +CPU= 49.436
  18:50:26 STEP  856 H= 214:00 +CPU= 57.118
  18:51:22 STEP  857 H= 214:15 +CPU= 54.727
  18:52:25 STEP  858 H= 214:30 +CPU= 61.925
  18:53:17 STEP  859 H= 214:45 +CPU= 51.427
  18:54:09 STEP  860 H= 215:00 +CPU= 51.882
  18:55:08 STEP  861 H= 215:15 +CPU= 58.667
  18:56:04 STEP  862 H= 215:30 +CPU= 54.793
  18:57:00 STEP  863 H= 215:45 +CPU= 56.092
  18:58:16 STEP  864 H= 216:00 +CPU= 75.150
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000864
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000864
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000852
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000852
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000852
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_8.zip
Uploading the intermediate file: upload_file_8.zip
  18:59:17 STEP  865 H= 216:15 +CPU= 59.361
  19:00:11 STEP  866 H= 216:30 +CPU= 53.896
  19:01:05 STEP  867 H= 216:45 +CPU= 53.615
  19:01:58 STEP  868 H= 217:00 +CPU= 52.329
  19:02:53 STEP  869 H= 217:15 +CPU= 54.661
  19:03:49 STEP  870 H= 217:30 +CPU= 55.759
  19:04:42 STEP  871 H= 217:45 +CPU= 51.796
  19:05:37 STEP  872 H= 218:00 +CPU= 55.734
  19:06:28 STEP  873 H= 218:15 +CPU= 50.070
  19:07:19 STEP  874 H= 218:30 +CPU= 50.763
  19:08:15 STEP  875 H= 218:45 +CPU= 55.501
  19:09:23 STEP  876 H= 219:00 +CPU= 67.332
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000876
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000876
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000876
  19:10:14 STEP  877 H= 219:15 +CPU= 49.897
  19:11:12 STEP  878 H= 219:30 +CPU= 57.358
  19:12:12 STEP  879 H= 219:45 +CPU= 59.577
  19:13:07 STEP  880 H= 220:00 +CPU= 54.719
  19:13:58 STEP  881 H= 220:15 +CPU= 51.055
  19:14:50 STEP  882 H= 220:30 +CPU= 50.798
  19:15:38 STEP  883 H= 220:45 +CPU= 47.280
  19:16:33 STEP  884 H= 221:00 +CPU= 54.258
  19:17:29 STEP  885 H= 221:15 +CPU= 54.468
  19:18:15 STEP  886 H= 221:30 +CPU= 45.202
  19:19:05 STEP  887 H= 221:45 +CPU= 50.151
  19:20:07 STEP  888 H= 222:00 +CPU= 59.545
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000888
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000888
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000888
  19:20:53 STEP  889 H= 222:15 +CPU= 45.584
  19:21:43 STEP  890 H= 222:30 +CPU= 49.587
  19:22:32 STEP  891 H= 222:45 +CPU= 47.697
  19:23:26 STEP  892 H= 223:00 +CPU= 53.928
  19:24:13 STEP  893 H= 223:15 +CPU= 46.317
  19:25:10 STEP  894 H= 223:30 +CPU= 56.141
  19:26:12 STEP  895 H= 223:45 +CPU= 61.312
  19:27:04 STEP  896 H= 224:00 +CPU= 51.166
  19:27:53 STEP  897 H= 224:15 +CPU= 48.368
  19:28:49 STEP  898 H= 224:30 +CPU= 55.772
  19:29:38 STEP  899 H= 224:45 +CPU= 47.650
  19:30:41 STEP  900 H= 225:00 +CPU= 61.969
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000900
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000900
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000900
  19:31:33 STEP  901 H= 225:15 +CPU= 51.381
  19:32:19 STEP  902 H= 225:30 +CPU= 46.012
  19:33:13 STEP  903 H= 225:45 +CPU= 53.758
  19:34:11 STEP  904 H= 226:00 +CPU= 56.490
  19:35:09 STEP  905 H= 226:15 +CPU= 57.943
  19:36:02 STEP  906 H= 226:30 +CPU= 52.700
  19:36:55 STEP  907 H= 226:45 +CPU= 51.374
  19:37:46 STEP  908 H= 227:00 +CPU= 50.834
  19:38:36 STEP  909 H= 227:15 +CPU= 49.361
  19:39:23 STEP  910 H= 227:30 +CPU= 46.060
  19:40:15 STEP  911 H= 227:45 +CPU= 51.762
  19:41:32 STEP  912 H= 228:00 +CPU= 75.791
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000912
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000912
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000912
  19:42:25 STEP  913 H= 228:15 +CPU= 53.222
  19:43:27 STEP  914 H= 228:30 +CPU= 61.457
  19:44:22 STEP  915 H= 228:45 +CPU= 54.214
  19:45:15 STEP  916 H= 229:00 +CPU= 52.541
  19:46:04 STEP  917 H= 229:15 +CPU= 49.506
  19:46:56 STEP  918 H= 229:30 +CPU= 50.728
  19:47:45 STEP  919 H= 229:45 +CPU= 49.320
  19:48:41 STEP  920 H= 230:00 +CPU= 55.304
  19:49:35 STEP  921 H= 230:15 +CPU= 53.864
  19:50:28 STEP  922 H= 230:30 +CPU= 52.276
  19:51:22 STEP  923 H= 230:45 +CPU= 53.381
  19:52:27 STEP  924 H= 231:00 +CPU= 65.002
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000924
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000924
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000924
  19:53:19 STEP  925 H= 231:15 +CPU= 50.987
  19:54:16 STEP  926 H= 231:30 +CPU= 56.095
  19:55:09 STEP  927 H= 231:45 +CPU= 53.271
  19:56:04 STEP  928 H= 232:00 +CPU= 54.145
  19:57:01 STEP  929 H= 232:15 +CPU= 56.279
  19:57:53 STEP  930 H= 232:30 +CPU= 52.449
  19:58:50 STEP  931 H= 232:45 +CPU= 56.572
  19:59:46 STEP  932 H= 233:00 +CPU= 55.841
  20:00:45 STEP  933 H= 233:15 +CPU= 58.131
  20:01:39 STEP  934 H= 233:30 +CPU= 53.331
  20:02:33 STEP  935 H= 233:45 +CPU= 53.597
  20:03:40 STEP  936 H= 234:00 +CPU= 66.639
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000936
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000936
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000936
  20:04:40 STEP  937 H= 234:15 +CPU= 59.585
  20:05:36 STEP  938 H= 234:30 +CPU= 55.689
  20:06:33 STEP  939 H= 234:45 +CPU= 57.046
  20:07:28 STEP  940 H= 235:00 +CPU= 54.762
  20:08:22 STEP  941 H= 235:15 +CPU= 53.073
  20:09:10 STEP  942 H= 235:30 +CPU= 47.644
  20:10:07 STEP  943 H= 235:45 +CPU= 56.214
  20:10:59 STEP  944 H= 236:00 +CPU= 51.638
  20:11:52 STEP  945 H= 236:15 +CPU= 52.266
  20:12:48 STEP  946 H= 236:30 +CPU= 55.737
  20:13:45 STEP  947 H= 236:45 +CPU= 55.766
  20:14:49 STEP  948 H= 237:00 +CPU= 63.304
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000948
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000948
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000948
  20:15:50 STEP  949 H= 237:15 +CPU= 59.802
  20:16:41 STEP  950 H= 237:30 +CPU= 50.939
  20:17:44 STEP  951 H= 237:45 +CPU= 62.573
  20:18:39 STEP  952 H= 238:00 +CPU= 54.846
  20:19:35 STEP  953 H= 238:15 +CPU= 55.287
  20:20:27 STEP  954 H= 238:30 +CPU= 52.060
  20:21:25 STEP  955 H= 238:45 +CPU= 56.254
  20:22:24 STEP  956 H= 239:00 +CPU= 59.162
  20:23:30 STEP  957 H= 239:15 +CPU= 65.167
  20:24:27 STEP  958 H= 239:30 +CPU= 56.427
  20:25:27 STEP  959 H= 239:45 +CPU= 59.913
  20:26:40 STEP  960 H= 240:00 +CPU= 72.108
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000960
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000960
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000948
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000948
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000948
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_9.zip
Uploading the intermediate file: upload_file_9.zip
  20:27:44 STEP  961 H= 240:15 +CPU= 62.915
  20:28:48 STEP  962 H= 240:30 +CPU= 63.619
  20:29:49 STEP  963 H= 240:45 +CPU= 61.580
  20:30:46 STEP  964 H= 241:00 +CPU= 56.145
  20:31:48 STEP  965 H= 241:15 +CPU= 61.829
  20:32:48 STEP  966 H= 241:30 +CPU= 59.105
  20:33:46 STEP  967 H= 241:45 +CPU= 57.750
  20:34:43 STEP  968 H= 242:00 +CPU= 55.832
  20:35:51 STEP  969 H= 242:15 +CPU= 67.854
  20:36:57 STEP  970 H= 242:30 +CPU= 65.284
  20:37:57 STEP  971 H= 242:45 +CPU= 59.718
  20:39:07 STEP  972 H= 243:00 +CPU= 69.411
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000972
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000972
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000972
  20:40:01 STEP  973 H= 243:15 +CPU= 54.033
  20:41:02 STEP  974 H= 243:30 +CPU= 59.821
  20:42:00 STEP  975 H= 243:45 +CPU= 57.065
  20:43:00 STEP  976 H= 244:00 +CPU= 60.501
  20:43:56 STEP  977 H= 244:15 +CPU= 55.098
  20:44:55 STEP  978 H= 244:30 +CPU= 58.226
  20:45:58 STEP  979 H= 244:45 +CPU= 62.663
  20:47:00 STEP  980 H= 245:00 +CPU= 60.970
  20:48:04 STEP  981 H= 245:15 +CPU= 63.281
  20:49:01 STEP  982 H= 245:30 +CPU= 56.306
  20:50:02 STEP  983 H= 245:45 +CPU= 61.015
  20:51:10 STEP  984 H= 246:00 +CPU= 66.454
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000984
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000984
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000984
  20:52:03 STEP  985 H= 246:15 +CPU= 52.187
  20:53:01 STEP  986 H= 246:30 +CPU= 57.676
  20:53:58 STEP  987 H= 246:45 +CPU= 55.804
  20:55:00 STEP  988 H= 247:00 +CPU= 61.649
  20:56:00 STEP  989 H= 247:15 +CPU= 58.842
  20:57:02 STEP  990 H= 247:30 +CPU= 61.826
  20:58:05 STEP  991 H= 247:45 +CPU= 61.849
  20:59:08 STEP  992 H= 248:00 +CPU= 62.814
  21:00:14 STEP  993 H= 248:15 +CPU= 64.935
  21:01:09 STEP  994 H= 248:30 +CPU= 54.450
  21:02:08 STEP  995 H= 248:45 +CPU= 58.594
  21:03:15 STEP  996 H= 249:00 +CPU= 65.758
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+000996
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+000996
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+000996
  21:04:13 STEP  997 H= 249:15 +CPU= 56.715
  21:05:11 STEP  998 H= 249:30 +CPU= 57.839
  21:06:09 STEP  999 H= 249:45 +CPU= 57.020
  21:07:05 STEP 1000 H= 250:00 +CPU= 56.293
  21:08:06 STEP 1001 H= 250:15 +CPU= 60.074
  21:09:04 STEP 1002 H= 250:30 +CPU= 57.184
  21:09:59 STEP 1003 H= 250:45 +CPU= 54.364
  21:10:59 STEP 1004 H= 251:00 +CPU= 58.697
  21:12:04 STEP 1005 H= 251:15 +CPU= 64.371
  21:13:06 STEP 1006 H= 251:30 +CPU= 60.426
  21:14:04 STEP 1007 H= 251:45 +CPU= 58.053
  21:15:29 STEP 1008 H= 252:00 +CPU= 82.447
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001008
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001008
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001008
  21:16:33 STEP 1009 H= 252:15 +CPU= 63.723
  21:17:32 STEP 1010 H= 252:30 +CPU= 58.674
  21:18:32 STEP 1011 H= 252:45 +CPU= 59.326
  21:19:39 STEP 1012 H= 253:00 +CPU= 66.087
  21:20:37 STEP 1013 H= 253:15 +CPU= 56.750
  21:21:38 STEP 1014 H= 253:30 +CPU= 61.055
  21:22:35 STEP 1015 H= 253:45 +CPU= 55.817
  21:23:30 STEP 1016 H= 254:00 +CPU= 53.409
  21:24:34 STEP 1017 H= 254:15 +CPU= 62.939
  21:25:36 STEP 1018 H= 254:30 +CPU= 61.552
  21:26:36 STEP 1019 H= 254:45 +CPU= 60.494
  21:27:50 STEP 1020 H= 255:00 +CPU= 73.035
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001020
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001020
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001020
  21:28:48 STEP 1021 H= 255:15 +CPU= 56.953
  21:29:56 STEP 1022 H= 255:30 +CPU= 66.798
  21:30:56 STEP 1023 H= 255:45 +CPU= 58.939
  21:31:59 STEP 1024 H= 256:00 +CPU= 62.837
  21:32:52 STEP 1025 H= 256:15 +CPU= 52.253
  21:33:51 STEP 1026 H= 256:30 +CPU= 57.455
  21:34:48 STEP 1027 H= 256:45 +CPU= 56.841
  21:35:52 STEP 1028 H= 257:00 +CPU= 63.373
  21:36:49 STEP 1029 H= 257:15 +CPU= 56.224
  21:37:48 STEP 1030 H= 257:30 +CPU= 58.940
  21:38:45 STEP 1031 H= 257:45 +CPU= 55.784
  21:39:55 STEP 1032 H= 258:00 +CPU= 69.935
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001032
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001032
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001032
  21:40:57 STEP 1033 H= 258:15 +CPU= 60.731
  21:41:55 STEP 1034 H= 258:30 +CPU= 57.512
  21:42:58 STEP 1035 H= 258:45 +CPU= 62.264
  21:44:01 STEP 1036 H= 259:00 +CPU= 62.200
  21:44:51 STEP 1037 H= 259:15 +CPU= 50.003
  21:45:43 STEP 1038 H= 259:30 +CPU= 51.412
  21:46:36 STEP 1039 H= 259:45 +CPU= 53.246
  21:47:41 STEP 1040 H= 260:00 +CPU= 63.567
  21:48:34 STEP 1041 H= 260:15 +CPU= 52.408
  21:49:31 STEP 1042 H= 260:30 +CPU= 56.782
  21:50:33 STEP 1043 H= 260:45 +CPU= 60.034
  21:51:40 STEP 1044 H= 261:00 +CPU= 66.089
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001044
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001044
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001044
  21:52:37 STEP 1045 H= 261:15 +CPU= 56.395
  21:53:44 STEP 1046 H= 261:30 +CPU= 65.433
  21:54:44 STEP 1047 H= 261:45 +CPU= 59.312
  21:55:44 STEP 1048 H= 262:00 +CPU= 59.314
  21:56:43 STEP 1049 H= 262:15 +CPU= 58.579
  21:57:47 STEP 1050 H= 262:30 +CPU= 62.969
  21:58:41 STEP 1051 H= 262:45 +CPU= 54.116
  21:59:44 STEP 1052 H= 263:00 +CPU= 61.794
  22:00:46 STEP 1053 H= 263:15 +CPU= 62.311
  22:01:54 STEP 1054 H= 263:30 +CPU= 66.521
  22:03:03 STEP 1055 H= 263:45 +CPU= 68.144
  22:04:18 STEP 1056 H= 264:00 +CPU= 74.217
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001056
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001056
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001044
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001044
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001044
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_10.zip
Uploading the intermediate file: upload_file_10.zip
  22:05:17 STEP 1057 H= 264:15 +CPU= 58.446
  22:06:17 STEP 1058 H= 264:30 +CPU= 59.120
  22:07:19 STEP 1059 H= 264:45 +CPU= 61.252
  22:08:21 STEP 1060 H= 265:00 +CPU= 60.372
  22:09:28 STEP 1061 H= 265:15 +CPU= 66.564
  22:10:32 STEP 1062 H= 265:30 +CPU= 62.492
  22:11:27 STEP 1063 H= 265:45 +CPU= 54.201
  22:12:34 STEP 1064 H= 266:00 +CPU= 66.148
  22:13:44 STEP 1065 H= 266:15 +CPU= 68.746
  22:14:47 STEP 1066 H= 266:30 +CPU= 61.666
  22:15:49 STEP 1067 H= 266:45 +CPU= 61.818
  22:17:07 STEP 1068 H= 267:00 +CPU= 76.872
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001068
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001068
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001068
  22:18:07 STEP 1069 H= 267:15 +CPU= 59.405
  22:19:11 STEP 1070 H= 267:30 +CPU= 63.739
  22:20:14 STEP 1071 H= 267:45 +CPU= 62.224
  22:21:16 STEP 1072 H= 268:00 +CPU= 61.332
  22:22:24 STEP 1073 H= 268:15 +CPU= 66.779
  22:23:25 STEP 1074 H= 268:30 +CPU= 61.271
  22:24:25 STEP 1075 H= 268:45 +CPU= 58.538
  22:25:23 STEP 1076 H= 269:00 +CPU= 57.053
  22:26:29 STEP 1077 H= 269:15 +CPU= 65.416
  22:27:35 STEP 1078 H= 269:30 +CPU= 65.205
  22:28:39 STEP 1079 H= 269:45 +CPU= 61.767
  22:29:53 STEP 1080 H= 270:00 +CPU= 72.854
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001080
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001080
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001080
  22:30:59 STEP 1081 H= 270:15 +CPU= 65.944
  22:32:07 STEP 1082 H= 270:30 +CPU= 66.414
  22:33:12 STEP 1083 H= 270:45 +CPU= 64.477
  22:34:23 STEP 1084 H= 271:00 +CPU= 69.905
  22:35:33 STEP 1085 H= 271:15 +CPU= 69.235
  22:36:41 STEP 1086 H= 271:30 +CPU= 67.731
  22:37:41 STEP 1087 H= 271:45 +CPU= 58.642
  22:38:44 STEP 1088 H= 272:00 +CPU= 62.554
  22:39:47 STEP 1089 H= 272:15 +CPU= 62.737
  22:40:56 STEP 1090 H= 272:30 +CPU= 67.720
  22:42:00 STEP 1091 H= 272:45 +CPU= 64.257
  22:43:18 STEP 1092 H= 273:00 +CPU= 77.347
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001092
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001092
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001092
  22:44:17 STEP 1093 H= 273:15 +CPU= 57.735
  22:45:26 STEP 1094 H= 273:30 +CPU= 69.208
  22:46:35 STEP 1095 H= 273:45 +CPU= 67.763
  22:47:42 STEP 1096 H= 274:00 +CPU= 66.361
  22:48:47 STEP 1097 H= 274:15 +CPU= 64.381
  22:49:45 STEP 1098 H= 274:30 +CPU= 57.365
  22:50:53 STEP 1099 H= 274:45 +CPU= 66.698
  22:52:03 STEP 1100 H= 275:00 +CPU= 68.304
  22:53:02 STEP 1101 H= 275:15 +CPU= 58.655
  22:54:08 STEP 1102 H= 275:30 +CPU= 64.612
  22:55:12 STEP 1103 H= 275:45 +CPU= 62.620
  22:56:42 STEP 1104 H= 276:00 +CPU= 88.436
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001104
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001104
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001104
  22:57:55 STEP 1105 H= 276:15 +CPU= 71.849
  22:58:58 STEP 1106 H= 276:30 +CPU= 62.774
  23:00:03 STEP 1107 H= 276:45 +CPU= 63.806
  23:01:06 STEP 1108 H= 277:00 +CPU= 61.860
  23:02:07 STEP 1109 H= 277:15 +CPU= 60.839
  23:03:14 STEP 1110 H= 277:30 +CPU= 66.795
  23:04:07 STEP 1111 H= 277:45 +CPU= 52.715
  23:05:11 STEP 1112 H= 278:00 +CPU= 62.935
  23:06:19 STEP 1113 H= 278:15 +CPU= 67.034
  23:07:18 STEP 1114 H= 278:30 +CPU= 58.340
  23:08:22 STEP 1115 H= 278:45 +CPU= 63.105
  23:09:37 STEP 1116 H= 279:00 +CPU= 73.950
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001116
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001116
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001116
  23:10:46 STEP 1117 H= 279:15 +CPU= 66.891
  23:12:00 STEP 1118 H= 279:30 +CPU= 72.661
  23:13:05 STEP 1119 H= 279:45 +CPU= 63.660
  23:13:55 STEP 1120 H= 280:00 +CPU= 50.049
  23:15:03 STEP 1121 H= 280:15 +CPU= 67.394
  23:16:09 STEP 1122 H= 280:30 +CPU= 65.051
  23:17:14 STEP 1123 H= 280:45 +CPU= 64.196
  23:18:26 STEP 1124 H= 281:00 +CPU= 70.981
  23:19:36 STEP 1125 H= 281:15 +CPU= 70.150
  23:20:42 STEP 1126 H= 281:30 +CPU= 64.481
  23:21:49 STEP 1127 H= 281:45 +CPU= 66.930
  23:23:04 STEP 1128 H= 282:00 +CPU= 72.962
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001128
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001128
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001128
  23:24:13 STEP 1129 H= 282:15 +CPU= 67.872
  23:25:15 STEP 1130 H= 282:30 +CPU= 61.048
  23:26:19 STEP 1131 H= 282:45 +CPU= 62.326
  23:27:26 STEP 1132 H= 283:00 +CPU= 64.948
  23:28:29 STEP 1133 H= 283:15 +CPU= 62.401
  23:29:35 STEP 1134 H= 283:30 +CPU= 65.824
  23:30:42 STEP 1135 H= 283:45 +CPU= 66.410
  23:31:41 STEP 1136 H= 284:00 +CPU= 58.361
  23:32:52 STEP 1137 H= 284:15 +CPU= 69.939
  23:34:01 STEP 1138 H= 284:30 +CPU= 68.496
  23:35:08 STEP 1139 H= 284:45 +CPU= 65.513
  23:36:24 STEP 1140 H= 285:00 +CPU= 75.153
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001140
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001140
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001140
  23:37:35 STEP 1141 H= 285:15 +CPU= 69.822
  23:38:37 STEP 1142 H= 285:30 +CPU= 61.007
  23:39:46 STEP 1143 H= 285:45 +CPU= 67.762
  23:40:48 STEP 1144 H= 286:00 +CPU= 61.520
  23:42:00 STEP 1145 H= 286:15 +CPU= 70.046
  23:43:00 STEP 1146 H= 286:30 +CPU= 59.765
  23:44:05 STEP 1147 H= 286:45 +CPU= 64.106
  23:45:14 STEP 1148 H= 287:00 +CPU= 67.213
  23:46:15 STEP 1149 H= 287:15 +CPU= 60.295
  23:47:25 STEP 1150 H= 287:30 +CPU= 68.523
  23:48:29 STEP 1151 H= 287:45 +CPU= 63.370
  23:49:51 STEP 1152 H= 288:00 +CPU= 80.958
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001152
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001152
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001140
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001140
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001140
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_11.zip
Uploading the intermediate file: upload_file_11.zip
  23:50:55 STEP 1153 H= 288:15 +CPU= 62.382
  23:51:59 STEP 1154 H= 288:30 +CPU= 62.793
  23:53:09 STEP 1155 H= 288:45 +CPU= 69.152
  23:54:18 STEP 1156 H= 289:00 +CPU= 67.748
  23:55:21 STEP 1157 H= 289:15 +CPU= 62.419
  23:56:20 STEP 1158 H= 289:30 +CPU= 57.455
  23:57:29 STEP 1159 H= 289:45 +CPU= 67.441
  23:58:34 STEP 1160 H= 290:00 +CPU= 64.665
  23:59:42 STEP 1161 H= 290:15 +CPU= 67.101
  00:00:48 STEP 1162 H= 290:30 +CPU= 64.582
  00:01:50 STEP 1163 H= 290:45 +CPU= 61.818
  00:03:07 STEP 1164 H= 291:00 +CPU= 75.748
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001164
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001164
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001164
  00:04:10 STEP 1165 H= 291:15 +CPU= 62.415
  00:05:22 STEP 1166 H= 291:30 +CPU= 70.663
  00:06:27 STEP 1167 H= 291:45 +CPU= 64.122
  00:07:36 STEP 1168 H= 292:00 +CPU= 68.921
  00:08:44 STEP 1169 H= 292:15 +CPU= 68.174
  00:09:50 STEP 1170 H= 292:30 +CPU= 64.730
  00:10:53 STEP 1171 H= 292:45 +CPU= 62.580
  00:12:03 STEP 1172 H= 293:00 +CPU= 67.953
  00:13:09 STEP 1173 H= 293:15 +CPU= 64.776
  00:14:12 STEP 1174 H= 293:30 +CPU= 62.014
  00:15:15 STEP 1175 H= 293:45 +CPU= 62.486
  00:16:36 STEP 1176 H= 294:00 +CPU= 80.071
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001176
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001176
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001176
  00:17:39 STEP 1177 H= 294:15 +CPU= 62.737
  00:18:42 STEP 1178 H= 294:30 +CPU= 61.204
  00:19:43 STEP 1179 H= 294:45 +CPU= 60.506
  00:20:41 STEP 1180 H= 295:00 +CPU= 57.478
  00:21:44 STEP 1181 H= 295:15 +CPU= 61.642
  00:22:43 STEP 1182 H= 295:30 +CPU= 59.150
  00:23:52 STEP 1183 H= 295:45 +CPU= 67.691
  00:24:56 STEP 1184 H= 296:00 +CPU= 64.405
  00:26:01 STEP 1185 H= 296:15 +CPU= 63.865
  00:26:59 STEP 1186 H= 296:30 +CPU= 57.576
  00:27:57 STEP 1187 H= 296:45 +CPU= 57.784
  00:29:18 STEP 1188 H= 297:00 +CPU= 79.469
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001188
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001188
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001188
  00:30:21 STEP 1189 H= 297:15 +CPU= 62.865
  00:31:27 STEP 1190 H= 297:30 +CPU= 64.196
  00:32:30 STEP 1191 H= 297:45 +CPU= 61.402
  00:33:40 STEP 1192 H= 298:00 +CPU= 69.336
  00:34:45 STEP 1193 H= 298:15 +CPU= 63.746
  00:35:50 STEP 1194 H= 298:30 +CPU= 65.151
  00:36:57 STEP 1195 H= 298:45 +CPU= 64.439
  00:38:02 STEP 1196 H= 299:00 +CPU= 63.581
  00:39:09 STEP 1197 H= 299:15 +CPU= 65.975
  00:40:08 STEP 1198 H= 299:30 +CPU= 57.664
  00:41:13 STEP 1199 H= 299:45 +CPU= 64.333
  00:42:42 STEP 1200 H= 300:00 +CPU= 88.363
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001200
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001200
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001200
  00:43:47 STEP 1201 H= 300:15 +CPU= 64.252
  00:44:43 STEP 1202 H= 300:30 +CPU= 55.787
  00:45:53 STEP 1203 H= 300:45 +CPU= 68.672
  00:46:54 STEP 1204 H= 301:00 +CPU= 60.653
  00:48:01 STEP 1205 H= 301:15 +CPU= 66.952
  00:49:12 STEP 1206 H= 301:30 +CPU= 69.776
  00:50:14 STEP 1207 H= 301:45 +CPU= 62.436
  00:51:24 STEP 1208 H= 302:00 +CPU= 68.669
  00:52:30 STEP 1209 H= 302:15 +CPU= 65.357
  00:53:31 STEP 1210 H= 302:30 +CPU= 60.818
  00:54:38 STEP 1211 H= 302:45 +CPU= 65.850
  00:55:54 STEP 1212 H= 303:00 +CPU= 75.232
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001212
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001212
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001212
  00:57:05 STEP 1213 H= 303:15 +CPU= 70.438
  00:58:08 STEP 1214 H= 303:30 +CPU= 62.532
  00:59:17 STEP 1215 H= 303:45 +CPU= 69.068
  01:00:27 STEP 1216 H= 304:00 +CPU= 69.545
  01:01:34 STEP 1217 H= 304:15 +CPU= 65.717
  01:02:36 STEP 1218 H= 304:30 +CPU= 62.625
  01:03:38 STEP 1219 H= 304:45 +CPU= 61.376
  01:04:45 STEP 1220 H= 305:00 +CPU= 66.449
  01:05:48 STEP 1221 H= 305:15 +CPU= 62.322
  01:07:03 STEP 1222 H= 305:30 +CPU= 74.811
  01:08:15 STEP 1223 H= 305:45 +CPU= 71.537
  01:09:28 STEP 1224 H= 306:00 +CPU= 72.828
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001224
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001224
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001224
  01:10:36 STEP 1225 H= 306:15 +CPU= 66.662
  01:11:45 STEP 1226 H= 306:30 +CPU= 68.263
  01:12:43 STEP 1227 H= 306:45 +CPU= 57.977
  01:13:55 STEP 1228 H= 307:00 +CPU= 70.385
  01:15:01 STEP 1229 H= 307:15 +CPU= 65.784
  01:16:18 STEP 1230 H= 307:30 +CPU= 76.245
  01:17:23 STEP 1231 H= 307:45 +CPU= 63.521
  01:18:31 STEP 1232 H= 308:00 +CPU= 67.646
  01:19:37 STEP 1233 H= 308:15 +CPU= 65.403
  01:20:38 STEP 1234 H= 308:30 +CPU= 60.548
  01:21:45 STEP 1235 H= 308:45 +CPU= 66.080
  01:23:04 STEP 1236 H= 309:00 +CPU= 76.549
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001236
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001236
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001236
  01:24:05 STEP 1237 H= 309:15 +CPU= 59.271
  01:25:14 STEP 1238 H= 309:30 +CPU= 67.443
  01:26:26 STEP 1239 H= 309:45 +CPU= 71.254
  01:27:31 STEP 1240 H= 310:00 +CPU= 64.286
  01:28:33 STEP 1241 H= 310:15 +CPU= 61.810
  01:29:36 STEP 1242 H= 310:30 +CPU= 62.087
  01:30:47 STEP 1243 H= 310:45 +CPU= 69.866
  01:31:57 STEP 1244 H= 311:00 +CPU= 69.483
  01:33:03 STEP 1245 H= 311:15 +CPU= 64.765
  01:34:03 STEP 1246 H= 311:30 +CPU= 59.934
  01:35:08 STEP 1247 H= 311:45 +CPU= 64.103
  01:36:41 STEP 1248 H= 312:00 +CPU= 91.367
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001248
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001248
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001236
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001236
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001236
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_12.zip
Uploading the intermediate file: upload_file_12.zip
  01:37:45 STEP 1249 H= 312:15 +CPU= 62.993
  01:38:51 STEP 1250 H= 312:30 +CPU= 63.987
  01:39:53 STEP 1251 H= 312:45 +CPU= 61.357
  01:41:01 STEP 1252 H= 313:00 +CPU= 67.219
  01:42:08 STEP 1253 H= 313:15 +CPU= 66.739
  01:43:17 STEP 1254 H= 313:30 +CPU= 66.963
  01:44:23 STEP 1255 H= 313:45 +CPU= 65.143
  01:45:19 STEP 1256 H= 314:00 +CPU= 55.747
  01:46:23 STEP 1257 H= 314:15 +CPU= 62.466
  01:47:32 STEP 1258 H= 314:30 +CPU= 68.087
  01:48:35 STEP 1259 H= 314:45 +CPU= 61.791
  01:49:43 STEP 1260 H= 315:00 +CPU= 67.257
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001260
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001260
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001260
  01:50:51 STEP 1261 H= 315:15 +CPU= 65.870
  01:51:58 STEP 1262 H= 315:30 +CPU= 64.935
  01:53:11 STEP 1263 H= 315:45 +CPU= 71.795
  01:54:17 STEP 1264 H= 316:00 +CPU= 64.793
  01:55:22 STEP 1265 H= 316:15 +CPU= 64.092
  01:56:24 STEP 1266 H= 316:30 +CPU= 61.722
  01:57:28 STEP 1267 H= 316:45 +CPU= 61.804
  01:58:35 STEP 1268 H= 317:00 +CPU= 66.149
  01:59:38 STEP 1269 H= 317:15 +CPU= 62.310
  02:00:45 STEP 1270 H= 317:30 +CPU= 66.491
  02:01:45 STEP 1271 H= 317:45 +CPU= 58.843
  02:02:59 STEP 1272 H= 318:00 +CPU= 73.583
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001272
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001272
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001272
  02:04:07 STEP 1273 H= 318:15 +CPU= 67.757
  02:05:12 STEP 1274 H= 318:30 +CPU= 64.029
  02:06:17 STEP 1275 H= 318:45 +CPU= 64.437
  02:07:22 STEP 1276 H= 319:00 +CPU= 64.546
  02:08:23 STEP 1277 H= 319:15 +CPU= 60.793
  02:09:29 STEP 1278 H= 319:30 +CPU= 63.987
  02:10:29 STEP 1279 H= 319:45 +CPU= 59.757
  02:11:25 STEP 1280 H= 320:00 +CPU= 55.863
  02:12:32 STEP 1281 H= 320:15 +CPU= 66.142
  02:13:29 STEP 1282 H= 320:30 +CPU= 56.677
  02:14:35 STEP 1283 H= 320:45 +CPU= 65.202
  02:15:51 STEP 1284 H= 321:00 +CPU= 75.246
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001284
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001284
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001284
  02:16:54 STEP 1285 H= 321:15 +CPU= 62.033
  02:17:55 STEP 1286 H= 321:30 +CPU= 60.368
  02:18:57 STEP 1287 H= 321:45 +CPU= 61.655
  02:19:58 STEP 1288 H= 322:00 +CPU= 61.072
  02:20:59 STEP 1289 H= 322:15 +CPU= 59.402
  02:22:05 STEP 1290 H= 322:30 +CPU= 65.659
  02:23:05 STEP 1291 H= 322:45 +CPU= 59.527
  02:24:01 STEP 1292 H= 323:00 +CPU= 54.817
  02:25:00 STEP 1293 H= 323:15 +CPU= 57.792
  02:26:01 STEP 1294 H= 323:30 +CPU= 59.638
  02:27:04 STEP 1295 H= 323:45 +CPU= 62.691
  02:28:34 STEP 1296 H= 324:00 +CPU= 89.453
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001296
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001296
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001296
  02:29:39 STEP 1297 H= 324:15 +CPU= 63.614
  02:30:30 STEP 1298 H= 324:30 +CPU= 51.209
  02:31:24 STEP 1299 H= 324:45 +CPU= 53.666
  02:32:15 STEP 1300 H= 325:00 +CPU= 50.065
  02:33:12 STEP 1301 H= 325:15 +CPU= 56.311
  02:34:04 STEP 1302 H= 325:30 +CPU= 51.406
  02:34:59 STEP 1303 H= 325:45 +CPU= 55.189
  02:36:02 STEP 1304 H= 326:00 +CPU= 62.108
  02:36:53 STEP 1305 H= 326:15 +CPU= 50.600
  02:37:48 STEP 1306 H= 326:30 +CPU= 54.031
  02:38:46 STEP 1307 H= 326:45 +CPU= 57.369
  02:39:58 STEP 1308 H= 327:00 +CPU= 70.919
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001308
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001308
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001308
  02:40:59 STEP 1309 H= 327:15 +CPU= 59.715
  02:41:53 STEP 1310 H= 327:30 +CPU= 52.404
  02:42:50 STEP 1311 H= 327:45 +CPU= 55.794
  02:43:51 STEP 1312 H= 328:00 +CPU= 59.890
  02:44:47 STEP 1313 H= 328:15 +CPU= 55.587
  02:45:40 STEP 1314 H= 328:30 +CPU= 52.114
  02:46:28 STEP 1315 H= 328:45 +CPU= 47.471
  02:47:25 STEP 1316 H= 329:00 +CPU= 56.618
  02:48:20 STEP 1317 H= 329:15 +CPU= 53.323
  02:49:13 STEP 1318 H= 329:30 +CPU= 52.407
  02:50:10 STEP 1319 H= 329:45 +CPU= 56.271
  02:51:18 STEP 1320 H= 330:00 +CPU= 65.846
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001320
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001320
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001320
  02:52:17 STEP 1321 H= 330:15 +CPU= 58.058
  02:53:17 STEP 1322 H= 330:30 +CPU= 58.018
  02:54:15 STEP 1323 H= 330:45 +CPU= 56.603
  02:55:00 STEP 1324 H= 331:00 +CPU= 44.707
  02:55:53 STEP 1325 H= 331:15 +CPU= 52.849
  02:56:54 STEP 1326 H= 331:30 +CPU= 60.363
  02:57:51 STEP 1327 H= 331:45 +CPU= 55.540
  02:58:49 STEP 1328 H= 332:00 +CPU= 57.083
  02:59:51 STEP 1329 H= 332:15 +CPU= 61.257
  03:00:45 STEP 1330 H= 332:30 +CPU= 53.648
  03:01:39 STEP 1331 H= 332:45 +CPU= 52.563
  03:02:43 STEP 1332 H= 333:00 +CPU= 63.650
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001332
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001332
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001332
  03:03:43 STEP 1333 H= 333:15 +CPU= 59.114
  03:04:35 STEP 1334 H= 333:30 +CPU= 51.079
  03:05:38 STEP 1335 H= 333:45 +CPU= 62.435
  03:06:32 STEP 1336 H= 334:00 +CPU= 52.512
  03:07:23 STEP 1337 H= 334:15 +CPU= 50.285
  03:08:22 STEP 1338 H= 334:30 +CPU= 58.166
  03:09:19 STEP 1339 H= 334:45 +CPU= 57.240
  03:10:22 STEP 1340 H= 335:00 +CPU= 61.294
  03:11:14 STEP 1341 H= 335:15 +CPU= 51.452
  03:12:13 STEP 1342 H= 335:30 +CPU= 57.890
  03:13:11 STEP 1343 H= 335:45 +CPU= 57.703
  03:14:26 STEP 1344 H= 336:00 +CPU= 73.470
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001344
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001344
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001332
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001332
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001332
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_13.zip
Uploading the intermediate file: upload_file_13.zip
  03:15:22 STEP 1345 H= 336:15 +CPU= 55.089
  03:16:18 STEP 1346 H= 336:30 +CPU= 55.190
  03:17:14 STEP 1347 H= 336:45 +CPU= 56.239
  03:18:08 STEP 1348 H= 337:00 +CPU= 53.273
  03:19:12 STEP 1349 H= 337:15 +CPU= 62.734
  03:20:06 STEP 1350 H= 337:30 +CPU= 54.015
  03:21:05 STEP 1351 H= 337:45 +CPU= 58.944
  03:22:10 STEP 1352 H= 338:00 +CPU= 63.853
  03:23:10 STEP 1353 H= 338:15 +CPU= 57.573
  03:24:05 STEP 1354 H= 338:30 +CPU= 54.190
  03:24:55 STEP 1355 H= 338:45 +CPU= 49.631
  03:25:54 STEP 1356 H= 339:00 +CPU= 58.640
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001356
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001356
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001356
  03:26:45 STEP 1357 H= 339:15 +CPU= 50.570
  03:27:41 STEP 1358 H= 339:30 +CPU= 55.314
  03:28:31 STEP 1359 H= 339:45 +CPU= 50.077
  03:29:29 STEP 1360 H= 340:00 +CPU= 56.939
  03:30:23 STEP 1361 H= 340:15 +CPU= 52.360
  03:31:19 STEP 1362 H= 340:30 +CPU= 55.464
  03:32:27 STEP 1363 H= 340:45 +CPU= 66.331
  03:33:23 STEP 1364 H= 341:00 +CPU= 55.646
  03:34:16 STEP 1365 H= 341:15 +CPU= 51.620
  03:35:15 STEP 1366 H= 341:30 +CPU= 57.101
  03:36:11 STEP 1367 H= 341:45 +CPU= 55.350
  03:37:17 STEP 1368 H= 342:00 +CPU= 64.210
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001368
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001368
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001368
  03:38:11 STEP 1369 H= 342:15 +CPU= 53.876
  03:39:08 STEP 1370 H= 342:30 +CPU= 56.658
  03:40:06 STEP 1371 H= 342:45 +CPU= 56.961
  03:41:06 STEP 1372 H= 343:00 +CPU= 59.111
  03:42:03 STEP 1373 H= 343:15 +CPU= 55.752
  03:42:57 STEP 1374 H= 343:30 +CPU= 52.671
  03:43:58 STEP 1375 H= 343:45 +CPU= 59.983
  03:44:57 STEP 1376 H= 344:00 +CPU= 58.226
  03:45:55 STEP 1377 H= 344:15 +CPU= 57.570
  03:46:58 STEP 1378 H= 344:30 +CPU= 61.148
  03:47:56 STEP 1379 H= 344:45 +CPU= 57.044
  03:49:11 STEP 1380 H= 345:00 +CPU= 73.590
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001380
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001380
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001380
  03:50:07 STEP 1381 H= 345:15 +CPU= 55.072
  03:51:02 STEP 1382 H= 345:30 +CPU= 54.606
  03:52:00 STEP 1383 H= 345:45 +CPU= 56.808
  03:52:58 STEP 1384 H= 346:00 +CPU= 56.594
  03:53:55 STEP 1385 H= 346:15 +CPU= 56.925
  03:54:51 STEP 1386 H= 346:30 +CPU= 54.302
  03:55:42 STEP 1387 H= 346:45 +CPU= 49.375
  03:56:41 STEP 1388 H= 347:00 +CPU= 57.651
  03:57:39 STEP 1389 H= 347:15 +CPU= 56.568
  03:58:36 STEP 1390 H= 347:30 +CPU= 56.318
  03:59:30 STEP 1391 H= 347:45 +CPU= 53.713
  04:00:45 STEP 1392 H= 348:00 +CPU= 74.442
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001392
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001392
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001392
  04:01:44 STEP 1393 H= 348:15 +CPU= 57.375
  04:02:37 STEP 1394 H= 348:30 +CPU= 52.688
  04:03:41 STEP 1395 H= 348:45 +CPU= 62.300
  04:04:38 STEP 1396 H= 349:00 +CPU= 57.279
  04:05:32 STEP 1397 H= 349:15 +CPU= 52.807
  04:06:26 STEP 1398 H= 349:30 +CPU= 53.930
  04:07:14 STEP 1399 H= 349:45 +CPU= 47.704
  04:08:07 STEP 1400 H= 350:00 +CPU= 52.403
  04:09:07 STEP 1401 H= 350:15 +CPU= 59.301
  04:10:03 STEP 1402 H= 350:30 +CPU= 55.163
  04:11:02 STEP 1403 H= 350:45 +CPU= 58.337
  04:12:18 STEP 1404 H= 351:00 +CPU= 74.578
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001404
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001404
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001404
  04:13:21 STEP 1405 H= 351:15 +CPU= 62.239
  04:14:15 STEP 1406 H= 351:30 +CPU= 52.815
  04:15:15 STEP 1407 H= 351:45 +CPU= 59.866
  04:16:19 STEP 1408 H= 352:00 +CPU= 62.645
  04:17:18 STEP 1409 H= 352:15 +CPU= 58.946
  04:18:14 STEP 1410 H= 352:30 +CPU= 56.229
  04:19:14 STEP 1411 H= 352:45 +CPU= 59.373
  04:20:15 STEP 1412 H= 353:00 +CPU= 60.470
  04:21:17 STEP 1413 H= 353:15 +CPU= 60.564
  04:22:15 STEP 1414 H= 353:30 +CPU= 56.891
  04:23:15 STEP 1415 H= 353:45 +CPU= 59.795
  04:24:18 STEP 1416 H= 354:00 +CPU= 62.829
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001416
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001416
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001416
  04:25:10 STEP 1417 H= 354:15 +CPU= 51.073
  04:26:11 STEP 1418 H= 354:30 +CPU= 59.922
  04:27:09 STEP 1419 H= 354:45 +CPU= 57.272
  04:27:59 STEP 1420 H= 355:00 +CPU= 49.492
  04:28:58 STEP 1421 H= 355:15 +CPU= 58.326
  04:29:53 STEP 1422 H= 355:30 +CPU= 54.873
  04:30:48 STEP 1423 H= 355:45 +CPU= 54.876
  04:31:53 STEP 1424 H= 356:00 +CPU= 63.820
  04:32:49 STEP 1425 H= 356:15 +CPU= 55.828
  04:33:48 STEP 1426 H= 356:30 +CPU= 57.291
  04:34:50 STEP 1427 H= 356:45 +CPU= 61.407
  04:35:58 STEP 1428 H= 357:00 +CPU= 67.758
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001428
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001428
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001428
  04:36:59 STEP 1429 H= 357:15 +CPU= 60.135
  04:37:58 STEP 1430 H= 357:30 +CPU= 58.096
  04:38:55 STEP 1431 H= 357:45 +CPU= 56.694
  04:39:57 STEP 1432 H= 358:00 +CPU= 60.806
  04:40:51 STEP 1433 H= 358:15 +CPU= 53.478
  04:41:49 STEP 1434 H= 358:30 +CPU= 56.675
  04:42:46 STEP 1435 H= 358:45 +CPU= 56.516
  04:43:40 STEP 1436 H= 359:00 +CPU= 53.893
  04:44:39 STEP 1437 H= 359:15 +CPU= 57.176
  04:45:29 STEP 1438 H= 359:30 +CPU= 49.464
  04:46:26 STEP 1439 H= 359:45 +CPU= 56.038
  04:47:42 STEP 1440 H= 360:00 +CPU= 75.580
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.5094062224E-21      I       0.1660743817E-05      I
ICloud fractionI      0.0000000000E+00      I       0.2283051987E-01      I
I  Relat. Hum. I      0.5803305970E-01      I       0.1401082560E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.2955937184E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    303973268 BYTES IN      7 RECORDS TRANSFERRED IN    0.0025 SECONDS121589.3 Mbytes/s, TOTAL TIME=    0.0199(12.6%)
 04:47:42 STEP 1440 H= 360:00 +CPU= 75.580
 END CNT3
 NSTEP =  1440 CNT0   000000000  55.766  55.766  56.270   0.991               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        22880/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    403    0       46      1044068    22880 ************    92.2     3.1    89.1     5.9   175.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.98      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  833   12034.4   12034.4     21.99      0.00
   9 SPCM     - SPECTRAL COMP.                     768    3012.8    3012.8      5.08      0.00
  10 SCAN2M - PHYSICS                              769   24336.1   24336.1     41.05      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 64    2174.5    2174.5      0.31      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 33     151.4     151.4      0.01      0.00
  17 GRIDFPOS IN CNT4                               64      63.5      63.5      0.01      0.00
  27 RESTART FILES - WRITING                        16    8698.0    8698.0      0.31      0.00
  28 RESTART FILES - READING                         1    8595.5    8595.5      0.02      0.00
  29 SU4FPOS IN CNT4                                64       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                64   10079.4   10079.4      1.42      0.00
  31 POSDDH IN STEPO                                 4       7.9       7.9      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    769   10987.8   10987.8     18.54      0.00
  39 SU0YOMB                                         1     919.6     919.6      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    769      68.1      68.1      0.11      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              768    1638.4    1638.4      2.76      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              768     155.1     155.1      0.26      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 33       1.1       1.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1668     764.2     764.2      2.80      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      1600     395.5     395.5      1.39      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       1600     179.6     179.6      0.63      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1668     266.3     266.3      0.97      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      71.3      71.3      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1668     176.6     176.6      0.65      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           1600     113.2     113.2      0.40      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1668     284.2     284.2      1.04      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           1600     139.7     139.7      0.49      0.00
 400 GSTATS                                      89902       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 83594       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    45585.4 CPU TIME   45050.9 VECTOR TIME    45050.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       28.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/test/slots/33/ICMGGh7zg+001440
Moving to projects directory: /home/test/slots/33/ICMSHh7zg+001440
Moving to projects directory: /home/test/slots/33/ICMUAh7zg+001440
Adding to the zip: /home/test/slots/33/NODE.001_01
Adding to the zip: /home/test/slots/33/ifs.stat
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMGGh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMUAh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282135/ICMSHh7zg+001344
Zipping up the final file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
04:51:14 (8429): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0_r109873265_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 10:08:48 1545820 22433702 oifs_43r3_bl_a08k_2016092300_20_1017_12282135_0 1,295,100 79,816 0.0616


©2024 cpdn.org