climateprediction.net home page
Task 22433725

Task 22433725

Name oifs_43r3_bl_a097_2016092300_20_1017_12282158_0
Workunit 12282158
Created 7 Jun 2024, 11:19:32 UTC
Sent 7 Jun 2024, 11:22:35 UTC
Report deadline 6 Aug 2024, 11:22:35 UTC
Received 8 Jun 2024, 10:08:46 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1545820
Run time 22 hours 19 min 10 sec
CPU time 22 hours 19 min 10 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 6.25 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.18 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,363.21 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
TEP  772 H= 193:00 +CPU= 55.162
  17:38:01 STEP  773 H= 193:15 +CPU= 50.174
  17:38:45 STEP  774 H= 193:30 +CPU= 43.948
  17:39:36 STEP  775 H= 193:45 +CPU= 50.960
  17:40:26 STEP  776 H= 194:00 +CPU= 49.133
  17:41:20 STEP  777 H= 194:15 +CPU= 54.111
  17:42:12 STEP  778 H= 194:30 +CPU= 52.115
  17:43:03 STEP  779 H= 194:45 +CPU= 49.950
  17:44:10 STEP  780 H= 195:00 +CPU= 67.360
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000780
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000780
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000780
  17:45:01 STEP  781 H= 195:15 +CPU= 49.717
  17:45:47 STEP  782 H= 195:30 +CPU= 46.505
  17:46:29 STEP  783 H= 195:45 +CPU= 41.847
  17:47:23 STEP  784 H= 196:00 +CPU= 52.875
  17:48:13 STEP  785 H= 196:15 +CPU= 50.126
  17:49:08 STEP  786 H= 196:30 +CPU= 54.697
  17:50:04 STEP  787 H= 196:45 +CPU= 55.321
  17:50:54 STEP  788 H= 197:00 +CPU= 49.442
  17:51:44 STEP  789 H= 197:15 +CPU= 49.243
  17:52:33 STEP  790 H= 197:30 +CPU= 49.459
  17:53:26 STEP  791 H= 197:45 +CPU= 52.955
  17:54:26 STEP  792 H= 198:00 +CPU= 59.106
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000792
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000792
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000792
  17:55:18 STEP  793 H= 198:15 +CPU= 51.424
  17:56:10 STEP  794 H= 198:30 +CPU= 52.011
  17:57:04 STEP  795 H= 198:45 +CPU= 53.844
  17:57:57 STEP  796 H= 199:00 +CPU= 52.042
  17:58:46 STEP  797 H= 199:15 +CPU= 47.785
  17:59:38 STEP  798 H= 199:30 +CPU= 52.357
  18:00:31 STEP  799 H= 199:45 +CPU= 52.710
  18:01:23 STEP  800 H= 200:00 +CPU= 51.157
  18:02:11 STEP  801 H= 200:15 +CPU= 48.340
  18:03:03 STEP  802 H= 200:30 +CPU= 51.009
  18:03:58 STEP  803 H= 200:45 +CPU= 55.059
  18:05:07 STEP  804 H= 201:00 +CPU= 68.846
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000804
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000804
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000804
  18:06:06 STEP  805 H= 201:15 +CPU= 58.249
  18:07:08 STEP  806 H= 201:30 +CPU= 61.973
  18:08:02 STEP  807 H= 201:45 +CPU= 52.944
  18:08:57 STEP  808 H= 202:00 +CPU= 54.837
  18:09:52 STEP  809 H= 202:15 +CPU= 54.083
  18:10:45 STEP  810 H= 202:30 +CPU= 52.898
  18:11:41 STEP  811 H= 202:45 +CPU= 54.866
  18:12:36 STEP  812 H= 203:00 +CPU= 54.865
  18:13:32 STEP  813 H= 203:15 +CPU= 54.926
  18:14:29 STEP  814 H= 203:30 +CPU= 56.428
  18:15:25 STEP  815 H= 203:45 +CPU= 55.508
  18:16:45 STEP  816 H= 204:00 +CPU= 79.549
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000816
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000816
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000816
  18:17:43 STEP  817 H= 204:15 +CPU= 58.059
  18:18:42 STEP  818 H= 204:30 +CPU= 57.741
  18:19:37 STEP  819 H= 204:45 +CPU= 54.664
  18:20:38 STEP  820 H= 205:00 +CPU= 59.890
  18:21:38 STEP  821 H= 205:15 +CPU= 60.138
  18:22:23 STEP  822 H= 205:30 +CPU= 44.325
  18:23:15 STEP  823 H= 205:45 +CPU= 52.048
  18:24:10 STEP  824 H= 206:00 +CPU= 53.572
  18:25:11 STEP  825 H= 206:15 +CPU= 61.313
  18:26:06 STEP  826 H= 206:30 +CPU= 53.923
  18:27:04 STEP  827 H= 206:45 +CPU= 58.012
  18:28:12 STEP  828 H= 207:00 +CPU= 67.397
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000828
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000828
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000828
  18:29:12 STEP  829 H= 207:15 +CPU= 59.281
  18:30:05 STEP  830 H= 207:30 +CPU= 52.660
  18:31:04 STEP  831 H= 207:45 +CPU= 58.496
  18:31:59 STEP  832 H= 208:00 +CPU= 54.583
  18:32:51 STEP  833 H= 208:15 +CPU= 52.231
  18:33:50 STEP  834 H= 208:30 +CPU= 58.134
  18:34:46 STEP  835 H= 208:45 +CPU= 55.346
  18:35:45 STEP  836 H= 209:00 +CPU= 58.558
  18:36:36 STEP  837 H= 209:15 +CPU= 50.745
  18:37:29 STEP  838 H= 209:30 +CPU= 52.328
  18:38:25 STEP  839 H= 209:45 +CPU= 54.946
  18:39:26 STEP  840 H= 210:00 +CPU= 61.177
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000840
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000840
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000840
  18:40:19 STEP  841 H= 210:15 +CPU= 52.471
  18:41:06 STEP  842 H= 210:30 +CPU= 46.646
  18:41:57 STEP  843 H= 210:45 +CPU= 50.165
  18:42:54 STEP  844 H= 211:00 +CPU= 56.521
  18:43:45 STEP  845 H= 211:15 +CPU= 51.439
  18:44:40 STEP  846 H= 211:30 +CPU= 53.601
  18:45:33 STEP  847 H= 211:45 +CPU= 52.310
  18:46:27 STEP  848 H= 212:00 +CPU= 53.374
  18:47:21 STEP  849 H= 212:15 +CPU= 54.188
  18:48:13 STEP  850 H= 212:30 +CPU= 50.960
  18:49:03 STEP  851 H= 212:45 +CPU= 49.671
  18:50:06 STEP  852 H= 213:00 +CPU= 62.122
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000852
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000852
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000852
  18:51:07 STEP  853 H= 213:15 +CPU= 60.977
  18:52:04 STEP  854 H= 213:30 +CPU= 56.538
  18:53:02 STEP  855 H= 213:45 +CPU= 57.217
  18:54:00 STEP  856 H= 214:00 +CPU= 56.308
  18:54:50 STEP  857 H= 214:15 +CPU= 50.105
  18:55:46 STEP  858 H= 214:30 +CPU= 55.664
  18:56:41 STEP  859 H= 214:45 +CPU= 54.974
  18:57:32 STEP  860 H= 215:00 +CPU= 50.407
  18:58:29 STEP  861 H= 215:15 +CPU= 56.727
  18:59:24 STEP  862 H= 215:30 +CPU= 54.386
  19:00:17 STEP  863 H= 215:45 +CPU= 52.319
  19:01:25 STEP  864 H= 216:00 +CPU= 67.150
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000864
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000864
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000768
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000780
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000792
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000804
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000816
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000828
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000840
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000852
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000852
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000852
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_8.zip
Uploading the intermediate file: upload_file_8.zip
  19:02:19 STEP  865 H= 216:15 +CPU= 54.318
  19:03:13 STEP  866 H= 216:30 +CPU= 52.967
  19:04:00 STEP  867 H= 216:45 +CPU= 46.679
  19:04:49 STEP  868 H= 217:00 +CPU= 48.900
  19:05:32 STEP  869 H= 217:15 +CPU= 42.628
  19:06:17 STEP  870 H= 217:30 +CPU= 44.791
  19:07:09 STEP  871 H= 217:45 +CPU= 50.546
  19:07:54 STEP  872 H= 218:00 +CPU= 45.225
  19:08:47 STEP  873 H= 218:15 +CPU= 52.522
  19:09:39 STEP  874 H= 218:30 +CPU= 51.427
  19:10:29 STEP  875 H= 218:45 +CPU= 49.451
  19:11:33 STEP  876 H= 219:00 +CPU= 63.826
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000876
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000876
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000876
  19:12:26 STEP  877 H= 219:15 +CPU= 52.146
  19:13:17 STEP  878 H= 219:30 +CPU= 50.474
  19:14:04 STEP  879 H= 219:45 +CPU= 46.772
  19:15:01 STEP  880 H= 220:00 +CPU= 57.042
  19:15:51 STEP  881 H= 220:15 +CPU= 49.028
  19:16:41 STEP  882 H= 220:30 +CPU= 49.314
  19:17:32 STEP  883 H= 220:45 +CPU= 48.910
  19:18:29 STEP  884 H= 221:00 +CPU= 56.662
  19:19:22 STEP  885 H= 221:15 +CPU= 52.708
  19:20:13 STEP  886 H= 221:30 +CPU= 50.877
  19:21:06 STEP  887 H= 221:45 +CPU= 52.457
  19:22:09 STEP  888 H= 222:00 +CPU= 62.055
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000888
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000888
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000888
  19:23:03 STEP  889 H= 222:15 +CPU= 53.053
  19:23:53 STEP  890 H= 222:30 +CPU= 49.535
  19:24:49 STEP  891 H= 222:45 +CPU= 55.143
  19:25:48 STEP  892 H= 223:00 +CPU= 57.644
  19:26:43 STEP  893 H= 223:15 +CPU= 54.512
  19:27:44 STEP  894 H= 223:30 +CPU= 60.147
  19:28:35 STEP  895 H= 223:45 +CPU= 50.483
  19:29:26 STEP  896 H= 224:00 +CPU= 50.210
  19:30:18 STEP  897 H= 224:15 +CPU= 51.564
  19:31:15 STEP  898 H= 224:30 +CPU= 56.577
  19:32:06 STEP  899 H= 224:45 +CPU= 50.490
  19:33:16 STEP  900 H= 225:00 +CPU= 68.834
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000900
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000900
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000900
  19:34:06 STEP  901 H= 225:15 +CPU= 49.778
  19:35:02 STEP  902 H= 225:30 +CPU= 55.103
  19:35:54 STEP  903 H= 225:45 +CPU= 51.988
  19:36:51 STEP  904 H= 226:00 +CPU= 56.307
  19:37:48 STEP  905 H= 226:15 +CPU= 56.040
  19:38:44 STEP  906 H= 226:30 +CPU= 54.790
  19:39:36 STEP  907 H= 226:45 +CPU= 50.555
  19:40:27 STEP  908 H= 227:00 +CPU= 50.357
  19:41:25 STEP  909 H= 227:15 +CPU= 57.980
  19:42:20 STEP  910 H= 227:30 +CPU= 54.004
  19:43:17 STEP  911 H= 227:45 +CPU= 56.203
  19:44:32 STEP  912 H= 228:00 +CPU= 73.926
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000912
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000912
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000912
  19:45:23 STEP  913 H= 228:15 +CPU= 50.221
  19:46:14 STEP  914 H= 228:30 +CPU= 50.050
  19:47:08 STEP  915 H= 228:45 +CPU= 52.553
  19:48:04 STEP  916 H= 229:00 +CPU= 56.227
  19:48:57 STEP  917 H= 229:15 +CPU= 52.648
  19:49:49 STEP  918 H= 229:30 +CPU= 50.551
  19:50:47 STEP  919 H= 229:45 +CPU= 58.096
  19:51:38 STEP  920 H= 230:00 +CPU= 50.313
  19:52:37 STEP  921 H= 230:15 +CPU= 57.545
  19:53:36 STEP  922 H= 230:30 +CPU= 58.089
  19:54:31 STEP  923 H= 230:45 +CPU= 54.448
  19:55:32 STEP  924 H= 231:00 +CPU= 60.399
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000924
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000924
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000924
  19:56:33 STEP  925 H= 231:15 +CPU= 60.559
  19:57:21 STEP  926 H= 231:30 +CPU= 47.865
  19:58:16 STEP  927 H= 231:45 +CPU= 55.297
  19:59:13 STEP  928 H= 232:00 +CPU= 56.105
  20:00:11 STEP  929 H= 232:15 +CPU= 57.187
  20:01:12 STEP  930 H= 232:30 +CPU= 60.501
  20:02:10 STEP  931 H= 232:45 +CPU= 58.046
  20:03:07 STEP  932 H= 233:00 +CPU= 56.952
  20:03:58 STEP  933 H= 233:15 +CPU= 50.370
  20:04:52 STEP  934 H= 233:30 +CPU= 52.622
  20:05:41 STEP  935 H= 233:45 +CPU= 49.434
  20:06:49 STEP  936 H= 234:00 +CPU= 66.572
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000936
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000936
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000936
  20:07:42 STEP  937 H= 234:15 +CPU= 52.569
  20:08:39 STEP  938 H= 234:30 +CPU= 56.460
  20:09:39 STEP  939 H= 234:45 +CPU= 59.858
  20:10:41 STEP  940 H= 235:00 +CPU= 60.411
  20:11:33 STEP  941 H= 235:15 +CPU= 51.685
  20:12:27 STEP  942 H= 235:30 +CPU= 53.776
  20:13:25 STEP  943 H= 235:45 +CPU= 56.056
  20:14:16 STEP  944 H= 236:00 +CPU= 50.850
  20:15:17 STEP  945 H= 236:15 +CPU= 60.731
  20:16:17 STEP  946 H= 236:30 +CPU= 60.106
  20:17:14 STEP  947 H= 236:45 +CPU= 57.052
  20:18:39 STEP  948 H= 237:00 +CPU= 83.875
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000948
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000948
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000948
  20:19:31 STEP  949 H= 237:15 +CPU= 51.673
  20:20:32 STEP  950 H= 237:30 +CPU= 59.521
  20:21:36 STEP  951 H= 237:45 +CPU= 63.786
  20:22:40 STEP  952 H= 238:00 +CPU= 62.713
  20:23:36 STEP  953 H= 238:15 +CPU= 56.235
  20:24:33 STEP  954 H= 238:30 +CPU= 55.548
  20:25:23 STEP  955 H= 238:45 +CPU= 49.725
  20:26:17 STEP  956 H= 239:00 +CPU= 53.928
  20:27:19 STEP  957 H= 239:15 +CPU= 60.550
  20:28:16 STEP  958 H= 239:30 +CPU= 56.509
  20:29:11 STEP  959 H= 239:45 +CPU= 54.305
  20:30:32 STEP  960 H= 240:00 +CPU= 80.756
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000960
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000960
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000864
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000876
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000888
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000900
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000912
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000924
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000936
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000948
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000948
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000948
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_9.zip
Uploading the intermediate file: upload_file_9.zip
  20:31:36 STEP  961 H= 240:15 +CPU= 63.694
  20:32:39 STEP  962 H= 240:30 +CPU= 61.895
  20:33:37 STEP  963 H= 240:45 +CPU= 57.808
  20:34:39 STEP  964 H= 241:00 +CPU= 61.118
  20:35:42 STEP  965 H= 241:15 +CPU= 62.099
  20:36:37 STEP  966 H= 241:30 +CPU= 55.049
  20:37:28 STEP  967 H= 241:45 +CPU= 50.184
  20:38:34 STEP  968 H= 242:00 +CPU= 66.005
  20:39:38 STEP  969 H= 242:15 +CPU= 62.971
  20:40:39 STEP  970 H= 242:30 +CPU= 60.176
  20:41:40 STEP  971 H= 242:45 +CPU= 60.089
  20:42:53 STEP  972 H= 243:00 +CPU= 72.433
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000972
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000972
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000972
  20:43:55 STEP  973 H= 243:15 +CPU= 61.678
  20:44:56 STEP  974 H= 243:30 +CPU= 60.731
  20:45:55 STEP  975 H= 243:45 +CPU= 57.949
  20:46:59 STEP  976 H= 244:00 +CPU= 63.951
  20:48:00 STEP  977 H= 244:15 +CPU= 59.952
  20:49:09 STEP  978 H= 244:30 +CPU= 68.094
  20:50:12 STEP  979 H= 244:45 +CPU= 62.934
  20:51:13 STEP  980 H= 245:00 +CPU= 60.839
  20:52:17 STEP  981 H= 245:15 +CPU= 62.625
  20:53:16 STEP  982 H= 245:30 +CPU= 59.375
  20:54:20 STEP  983 H= 245:45 +CPU= 62.546
  20:55:36 STEP  984 H= 246:00 +CPU= 75.607
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000984
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000984
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000984
  20:56:36 STEP  985 H= 246:15 +CPU= 60.144
  20:57:37 STEP  986 H= 246:30 +CPU= 60.144
  20:58:40 STEP  987 H= 246:45 +CPU= 62.917
  20:59:42 STEP  988 H= 247:00 +CPU= 61.655
  21:00:43 STEP  989 H= 247:15 +CPU= 60.483
  21:01:40 STEP  990 H= 247:30 +CPU= 56.498
  21:02:41 STEP  991 H= 247:45 +CPU= 60.754
  21:03:48 STEP  992 H= 248:00 +CPU= 66.534
  21:04:49 STEP  993 H= 248:15 +CPU= 60.097
  21:05:50 STEP  994 H= 248:30 +CPU= 60.471
  21:06:52 STEP  995 H= 248:45 +CPU= 60.394
  21:07:59 STEP  996 H= 249:00 +CPU= 66.497
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+000996
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+000996
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+000996
  21:09:03 STEP  997 H= 249:15 +CPU= 62.893
  21:10:02 STEP  998 H= 249:30 +CPU= 58.716
  21:10:59 STEP  999 H= 249:45 +CPU= 55.665
  21:11:58 STEP 1000 H= 250:00 +CPU= 58.539
  21:12:59 STEP 1001 H= 250:15 +CPU= 59.648
  21:14:01 STEP 1002 H= 250:30 +CPU= 60.919
  21:15:05 STEP 1003 H= 250:45 +CPU= 62.818
  21:16:13 STEP 1004 H= 251:00 +CPU= 66.787
  21:17:16 STEP 1005 H= 251:15 +CPU= 61.752
  21:18:16 STEP 1006 H= 251:30 +CPU= 59.888
  21:19:16 STEP 1007 H= 251:45 +CPU= 59.380
  21:20:48 STEP 1008 H= 252:00 +CPU= 90.629
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001008
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001008
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001008
  21:21:48 STEP 1009 H= 252:15 +CPU= 58.457
  21:22:49 STEP 1010 H= 252:30 +CPU= 60.470
  21:23:50 STEP 1011 H= 252:45 +CPU= 60.620
  21:24:52 STEP 1012 H= 253:00 +CPU= 60.982
  21:25:50 STEP 1013 H= 253:15 +CPU= 57.606
  21:26:52 STEP 1014 H= 253:30 +CPU= 60.532
  21:27:46 STEP 1015 H= 253:45 +CPU= 53.279
  21:28:48 STEP 1016 H= 254:00 +CPU= 61.405
  21:29:46 STEP 1017 H= 254:15 +CPU= 58.115
  21:30:44 STEP 1018 H= 254:30 +CPU= 57.391
  21:31:42 STEP 1019 H= 254:45 +CPU= 57.322
  21:32:57 STEP 1020 H= 255:00 +CPU= 73.042
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001020
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001020
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001020
  21:33:50 STEP 1021 H= 255:15 +CPU= 53.059
  21:34:49 STEP 1022 H= 255:30 +CPU= 58.681
  21:35:49 STEP 1023 H= 255:45 +CPU= 58.869
  21:36:53 STEP 1024 H= 256:00 +CPU= 63.283
  21:37:55 STEP 1025 H= 256:15 +CPU= 61.209
  21:38:54 STEP 1026 H= 256:30 +CPU= 58.250
  21:40:00 STEP 1027 H= 256:45 +CPU= 64.428
  21:41:03 STEP 1028 H= 257:00 +CPU= 62.491
  21:42:02 STEP 1029 H= 257:15 +CPU= 58.438
  21:43:05 STEP 1030 H= 257:30 +CPU= 61.951
  21:44:05 STEP 1031 H= 257:45 +CPU= 59.295
  21:45:18 STEP 1032 H= 258:00 +CPU= 71.438
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001032
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001032
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001032
  21:46:16 STEP 1033 H= 258:15 +CPU= 56.843
  21:47:10 STEP 1034 H= 258:30 +CPU= 53.706
  21:48:12 STEP 1035 H= 258:45 +CPU= 61.007
  21:49:09 STEP 1036 H= 259:00 +CPU= 56.833
  21:50:14 STEP 1037 H= 259:15 +CPU= 62.614
  21:51:20 STEP 1038 H= 259:30 +CPU= 65.994
  21:52:19 STEP 1039 H= 259:45 +CPU= 58.374
  21:53:18 STEP 1040 H= 260:00 +CPU= 58.861
  21:54:14 STEP 1041 H= 260:15 +CPU= 55.251
  21:55:05 STEP 1042 H= 260:30 +CPU= 50.620
  21:56:07 STEP 1043 H= 260:45 +CPU= 60.011
  21:57:23 STEP 1044 H= 261:00 +CPU= 76.133
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001044
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001044
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001044
  21:58:28 STEP 1045 H= 261:15 +CPU= 63.402
  21:59:32 STEP 1046 H= 261:30 +CPU= 62.444
  22:00:32 STEP 1047 H= 261:45 +CPU= 60.006
  22:01:29 STEP 1048 H= 262:00 +CPU= 56.199
  22:02:32 STEP 1049 H= 262:15 +CPU= 62.067
  22:03:34 STEP 1050 H= 262:30 +CPU= 62.140
  22:04:38 STEP 1051 H= 262:45 +CPU= 62.473
  22:05:37 STEP 1052 H= 263:00 +CPU= 58.894
  22:06:46 STEP 1053 H= 263:15 +CPU= 67.927
  22:07:55 STEP 1054 H= 263:30 +CPU= 68.728
  22:08:57 STEP 1055 H= 263:45 +CPU= 60.387
  22:10:23 STEP 1056 H= 264:00 +CPU= 85.185
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001056
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001056
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000960
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000972
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000984
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+000996
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001008
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001020
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001032
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001044
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001044
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001044
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_10.zip
Uploading the intermediate file: upload_file_10.zip
  22:11:25 STEP 1057 H= 264:15 +CPU= 61.552
  22:12:26 STEP 1058 H= 264:30 +CPU= 59.502
  22:13:26 STEP 1059 H= 264:45 +CPU= 59.343
  22:14:32 STEP 1060 H= 265:00 +CPU= 65.407
  22:15:40 STEP 1061 H= 265:15 +CPU= 67.095
  22:16:40 STEP 1062 H= 265:30 +CPU= 59.899
  22:17:49 STEP 1063 H= 265:45 +CPU= 68.028
  22:19:00 STEP 1064 H= 266:00 +CPU= 70.395
  22:20:01 STEP 1065 H= 266:15 +CPU= 60.711
  22:21:03 STEP 1066 H= 266:30 +CPU= 61.363
  22:22:05 STEP 1067 H= 266:45 +CPU= 61.741
  22:23:28 STEP 1068 H= 267:00 +CPU= 81.858
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001068
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001068
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001068
  22:24:26 STEP 1069 H= 267:15 +CPU= 57.001
  22:25:29 STEP 1070 H= 267:30 +CPU= 62.734
  22:26:24 STEP 1071 H= 267:45 +CPU= 54.019
  22:27:27 STEP 1072 H= 268:00 +CPU= 62.331
  22:28:31 STEP 1073 H= 268:15 +CPU= 63.269
  22:29:43 STEP 1074 H= 268:30 +CPU= 70.660
  22:30:53 STEP 1075 H= 268:45 +CPU= 68.807
  22:31:55 STEP 1076 H= 269:00 +CPU= 61.088
  22:32:59 STEP 1077 H= 269:15 +CPU= 62.774
  22:34:03 STEP 1078 H= 269:30 +CPU= 63.006
  22:35:10 STEP 1079 H= 269:45 +CPU= 65.038
  22:36:30 STEP 1080 H= 270:00 +CPU= 79.664
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001080
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001080
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001080
  22:37:35 STEP 1081 H= 270:15 +CPU= 64.287
  22:38:39 STEP 1082 H= 270:30 +CPU= 64.164
  22:39:43 STEP 1083 H= 270:45 +CPU= 63.539
  22:40:40 STEP 1084 H= 271:00 +CPU= 55.868
  22:41:40 STEP 1085 H= 271:15 +CPU= 60.697
  22:42:41 STEP 1086 H= 271:30 +CPU= 59.992
  22:43:45 STEP 1087 H= 271:45 +CPU= 62.952
  22:44:51 STEP 1088 H= 272:00 +CPU= 65.017
  22:45:56 STEP 1089 H= 272:15 +CPU= 65.225
  22:47:02 STEP 1090 H= 272:30 +CPU= 64.594
  22:48:08 STEP 1091 H= 272:45 +CPU= 65.548
  22:49:22 STEP 1092 H= 273:00 +CPU= 72.110
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001092
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001092
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001092
  22:50:28 STEP 1093 H= 273:15 +CPU= 65.083
  22:51:29 STEP 1094 H= 273:30 +CPU= 58.416
  22:52:32 STEP 1095 H= 273:45 +CPU= 61.278
  22:53:39 STEP 1096 H= 274:00 +CPU= 65.826
  22:54:42 STEP 1097 H= 274:15 +CPU= 62.253
  22:55:46 STEP 1098 H= 274:30 +CPU= 63.282
  22:56:56 STEP 1099 H= 274:45 +CPU= 69.026
  22:58:00 STEP 1100 H= 275:00 +CPU= 62.933
  22:59:03 STEP 1101 H= 275:15 +CPU= 62.184
  23:00:06 STEP 1102 H= 275:30 +CPU= 62.980
  23:01:06 STEP 1103 H= 275:45 +CPU= 59.607
  23:02:40 STEP 1104 H= 276:00 +CPU= 92.637
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001104
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001104
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001104
  23:03:47 STEP 1105 H= 276:15 +CPU= 66.960
  23:04:53 STEP 1106 H= 276:30 +CPU= 65.210
  23:06:01 STEP 1107 H= 276:45 +CPU= 66.927
  23:07:05 STEP 1108 H= 277:00 +CPU= 63.224
  23:08:18 STEP 1109 H= 277:15 +CPU= 71.137
  23:09:29 STEP 1110 H= 277:30 +CPU= 69.774
  23:10:38 STEP 1111 H= 277:45 +CPU= 68.355
  23:11:42 STEP 1112 H= 278:00 +CPU= 62.732
  23:12:45 STEP 1113 H= 278:15 +CPU= 62.641
  23:13:42 STEP 1114 H= 278:30 +CPU= 56.475
  23:14:38 STEP 1115 H= 278:45 +CPU= 55.638
  23:15:54 STEP 1116 H= 279:00 +CPU= 76.257
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001116
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001116
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001116
  23:16:58 STEP 1117 H= 279:15 +CPU= 62.739
  23:17:58 STEP 1118 H= 279:30 +CPU= 60.122
  23:18:52 STEP 1119 H= 279:45 +CPU= 54.196
  23:19:51 STEP 1120 H= 280:00 +CPU= 58.329
  23:20:59 STEP 1121 H= 280:15 +CPU= 66.809
  23:22:02 STEP 1122 H= 280:30 +CPU= 62.687
  23:23:05 STEP 1123 H= 280:45 +CPU= 60.511
  23:24:05 STEP 1124 H= 281:00 +CPU= 59.202
  23:25:04 STEP 1125 H= 281:15 +CPU= 59.324
  23:26:02 STEP 1126 H= 281:30 +CPU= 57.394
  23:27:07 STEP 1127 H= 281:45 +CPU= 63.914
  23:28:25 STEP 1128 H= 282:00 +CPU= 77.054
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001128
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001128
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001128
  23:29:29 STEP 1129 H= 282:15 +CPU= 63.597
  23:30:29 STEP 1130 H= 282:30 +CPU= 59.553
  23:31:38 STEP 1131 H= 282:45 +CPU= 67.582
  23:32:50 STEP 1132 H= 283:00 +CPU= 71.958
  23:34:01 STEP 1133 H= 283:15 +CPU= 70.426
  23:35:02 STEP 1134 H= 283:30 +CPU= 59.679
  23:35:59 STEP 1135 H= 283:45 +CPU= 56.558
  23:37:02 STEP 1136 H= 284:00 +CPU= 63.035
  23:38:08 STEP 1137 H= 284:15 +CPU= 64.538
  23:39:10 STEP 1138 H= 284:30 +CPU= 62.014
  23:40:16 STEP 1139 H= 284:45 +CPU= 64.557
  23:41:41 STEP 1140 H= 285:00 +CPU= 84.030
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001140
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001140
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001140
  23:42:46 STEP 1141 H= 285:15 +CPU= 64.016
  23:43:51 STEP 1142 H= 285:30 +CPU= 64.038
  23:45:02 STEP 1143 H= 285:45 +CPU= 69.999
  23:46:07 STEP 1144 H= 286:00 +CPU= 63.171
  23:47:11 STEP 1145 H= 286:15 +CPU= 62.530
  23:48:17 STEP 1146 H= 286:30 +CPU= 65.327
  23:49:25 STEP 1147 H= 286:45 +CPU= 67.287
  23:50:37 STEP 1148 H= 287:00 +CPU= 69.390
  23:51:40 STEP 1149 H= 287:15 +CPU= 62.854
  23:52:49 STEP 1150 H= 287:30 +CPU= 67.630
  23:53:58 STEP 1151 H= 287:45 +CPU= 67.698
  23:55:30 STEP 1152 H= 288:00 +CPU= 90.740
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001152
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001152
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001056
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001068
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001080
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001092
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001104
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001116
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001128
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001140
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001140
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001140
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_11.zip
Uploading the intermediate file: upload_file_11.zip
  23:56:30 STEP 1153 H= 288:15 +CPU= 59.012
  23:57:29 STEP 1154 H= 288:30 +CPU= 58.531
  23:58:31 STEP 1155 H= 288:45 +CPU= 61.843
  23:59:36 STEP 1156 H= 289:00 +CPU= 63.821
  00:00:43 STEP 1157 H= 289:15 +CPU= 66.449
  00:01:46 STEP 1158 H= 289:30 +CPU= 63.462
  00:02:45 STEP 1159 H= 289:45 +CPU= 58.374
  00:03:50 STEP 1160 H= 290:00 +CPU= 64.725
  00:04:58 STEP 1161 H= 290:15 +CPU= 66.868
  00:06:06 STEP 1162 H= 290:30 +CPU= 67.625
  00:07:01 STEP 1163 H= 290:45 +CPU= 55.131
  00:08:17 STEP 1164 H= 291:00 +CPU= 74.174
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001164
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001164
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001164
  00:09:19 STEP 1165 H= 291:15 +CPU= 61.259
  00:10:22 STEP 1166 H= 291:30 +CPU= 62.466
  00:11:28 STEP 1167 H= 291:45 +CPU= 64.788
  00:12:35 STEP 1168 H= 292:00 +CPU= 65.117
  00:13:49 STEP 1169 H= 292:15 +CPU= 73.286
  00:14:53 STEP 1170 H= 292:30 +CPU= 64.070
  00:16:01 STEP 1171 H= 292:45 +CPU= 66.913
  00:17:06 STEP 1172 H= 293:00 +CPU= 64.718
  00:18:09 STEP 1173 H= 293:15 +CPU= 62.092
  00:19:18 STEP 1174 H= 293:30 +CPU= 67.582
  00:20:26 STEP 1175 H= 293:45 +CPU= 67.311
  00:21:42 STEP 1176 H= 294:00 +CPU= 75.130
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001176
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001176
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001176
  00:22:45 STEP 1177 H= 294:15 +CPU= 62.976
  00:23:54 STEP 1178 H= 294:30 +CPU= 67.027
  00:24:56 STEP 1179 H= 294:45 +CPU= 62.032
  00:25:58 STEP 1180 H= 295:00 +CPU= 61.005
  00:27:03 STEP 1181 H= 295:15 +CPU= 65.200
  00:28:01 STEP 1182 H= 295:30 +CPU= 57.116
  00:29:00 STEP 1183 H= 295:45 +CPU= 58.876
  00:30:05 STEP 1184 H= 296:00 +CPU= 64.276
  00:31:11 STEP 1185 H= 296:15 +CPU= 63.720
  00:32:14 STEP 1186 H= 296:30 +CPU= 62.128
  00:33:10 STEP 1187 H= 296:45 +CPU= 56.362
  00:34:35 STEP 1188 H= 297:00 +CPU= 82.972
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001188
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001188
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001188
  00:35:35 STEP 1189 H= 297:15 +CPU= 59.818
  00:36:40 STEP 1190 H= 297:30 +CPU= 64.178
  00:37:47 STEP 1191 H= 297:45 +CPU= 65.032
  00:38:48 STEP 1192 H= 298:00 +CPU= 60.768
  00:39:56 STEP 1193 H= 298:15 +CPU= 66.389
  00:40:59 STEP 1194 H= 298:30 +CPU= 62.486
  00:41:58 STEP 1195 H= 298:45 +CPU= 58.328
  00:43:09 STEP 1196 H= 299:00 +CPU= 69.866
  00:44:15 STEP 1197 H= 299:15 +CPU= 66.173
  00:45:17 STEP 1198 H= 299:30 +CPU= 60.616
  00:46:25 STEP 1199 H= 299:45 +CPU= 67.284
  00:47:52 STEP 1200 H= 300:00 +CPU= 86.362
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001200
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001200
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001200
  00:49:03 STEP 1201 H= 300:15 +CPU= 69.626
  00:50:12 STEP 1202 H= 300:30 +CPU= 68.279
  00:51:21 STEP 1203 H= 300:45 +CPU= 69.122
  00:52:29 STEP 1204 H= 301:00 +CPU= 66.783
  00:53:34 STEP 1205 H= 301:15 +CPU= 64.605
  00:54:40 STEP 1206 H= 301:30 +CPU= 66.336
  00:55:48 STEP 1207 H= 301:45 +CPU= 66.641
  00:56:59 STEP 1208 H= 302:00 +CPU= 71.058
  00:58:07 STEP 1209 H= 302:15 +CPU= 66.374
  00:59:18 STEP 1210 H= 302:30 +CPU= 70.052
  01:00:23 STEP 1211 H= 302:45 +CPU= 64.592
  01:01:46 STEP 1212 H= 303:00 +CPU= 82.940
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001212
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001212
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001212
  01:02:57 STEP 1213 H= 303:15 +CPU= 70.644
  01:04:04 STEP 1214 H= 303:30 +CPU= 66.040
  01:05:11 STEP 1215 H= 303:45 +CPU= 66.790
  01:06:16 STEP 1216 H= 304:00 +CPU= 64.272
  01:07:20 STEP 1217 H= 304:15 +CPU= 62.961
  01:08:28 STEP 1218 H= 304:30 +CPU= 67.108
  01:09:37 STEP 1219 H= 304:45 +CPU= 67.618
  01:10:50 STEP 1220 H= 305:00 +CPU= 72.611
  01:11:58 STEP 1221 H= 305:15 +CPU= 66.578
  01:13:03 STEP 1222 H= 305:30 +CPU= 64.994
  01:14:12 STEP 1223 H= 305:45 +CPU= 67.530
  01:15:34 STEP 1224 H= 306:00 +CPU= 81.918
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001224
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001224
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001224
  01:16:38 STEP 1225 H= 306:15 +CPU= 63.664
  01:17:48 STEP 1226 H= 306:30 +CPU= 68.034
  01:18:57 STEP 1227 H= 306:45 +CPU= 69.185
  01:20:09 STEP 1228 H= 307:00 +CPU= 71.644
  01:21:20 STEP 1229 H= 307:15 +CPU= 69.620
  01:22:24 STEP 1230 H= 307:30 +CPU= 63.574
  01:23:35 STEP 1231 H= 307:45 +CPU= 69.964
  01:24:40 STEP 1232 H= 308:00 +CPU= 64.136
  01:25:44 STEP 1233 H= 308:15 +CPU= 63.221
  01:26:52 STEP 1234 H= 308:30 +CPU= 66.792
  01:27:56 STEP 1235 H= 308:45 +CPU= 64.334
  01:29:13 STEP 1236 H= 309:00 +CPU= 75.216
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001236
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001236
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001236
  01:30:21 STEP 1237 H= 309:15 +CPU= 67.264
  01:31:28 STEP 1238 H= 309:30 +CPU= 66.385
  01:32:28 STEP 1239 H= 309:45 +CPU= 59.379
  01:33:36 STEP 1240 H= 310:00 +CPU= 66.564
  01:34:39 STEP 1241 H= 310:15 +CPU= 62.644
  01:35:47 STEP 1242 H= 310:30 +CPU= 67.643
  01:37:01 STEP 1243 H= 310:45 +CPU= 71.741
  01:38:06 STEP 1244 H= 311:00 +CPU= 63.979
  01:39:16 STEP 1245 H= 311:15 +CPU= 68.527
  01:40:23 STEP 1246 H= 311:30 +CPU= 66.240
  01:41:27 STEP 1247 H= 311:45 +CPU= 63.863
  01:42:58 STEP 1248 H= 312:00 +CPU= 89.931
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001248
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001248
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001152
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001164
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001176
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001188
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001200
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001212
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001224
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001236
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001236
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001236
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_12.zip
Uploading the intermediate file: upload_file_12.zip
  01:43:58 STEP 1249 H= 312:15 +CPU= 58.889
  01:45:10 STEP 1250 H= 312:30 +CPU= 70.416
  01:46:14 STEP 1251 H= 312:45 +CPU= 62.628
  01:47:23 STEP 1252 H= 313:00 +CPU= 69.350
  01:48:29 STEP 1253 H= 313:15 +CPU= 64.644
  01:49:40 STEP 1254 H= 313:30 +CPU= 70.315
  01:50:51 STEP 1255 H= 313:45 +CPU= 68.802
  01:52:00 STEP 1256 H= 314:00 +CPU= 67.278
  01:53:15 STEP 1257 H= 314:15 +CPU= 73.404
  01:54:21 STEP 1258 H= 314:30 +CPU= 65.532
  01:55:29 STEP 1259 H= 314:45 +CPU= 66.782
  01:56:47 STEP 1260 H= 315:00 +CPU= 74.706
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001260
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001260
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001260
  01:57:48 STEP 1261 H= 315:15 +CPU= 60.125
  01:58:43 STEP 1262 H= 315:30 +CPU= 53.531
  01:59:49 STEP 1263 H= 315:45 +CPU= 65.357
  02:00:57 STEP 1264 H= 316:00 +CPU= 66.803
  02:02:08 STEP 1265 H= 316:15 +CPU= 70.715
  02:03:12 STEP 1266 H= 316:30 +CPU= 63.117
  02:04:11 STEP 1267 H= 316:45 +CPU= 58.271
  02:05:18 STEP 1268 H= 317:00 +CPU= 66.786
  02:06:18 STEP 1269 H= 317:15 +CPU= 58.903
  02:07:20 STEP 1270 H= 317:30 +CPU= 61.292
  02:08:24 STEP 1271 H= 317:45 +CPU= 63.974
  02:09:39 STEP 1272 H= 318:00 +CPU= 73.947
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001272
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001272
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001272
  02:10:38 STEP 1273 H= 318:15 +CPU= 58.262
  02:11:48 STEP 1274 H= 318:30 +CPU= 68.936
  02:12:48 STEP 1275 H= 318:45 +CPU= 58.808
  02:13:54 STEP 1276 H= 319:00 +CPU= 64.850
  02:14:56 STEP 1277 H= 319:15 +CPU= 60.549
  02:16:00 STEP 1278 H= 319:30 +CPU= 62.838
  02:17:08 STEP 1279 H= 319:45 +CPU= 66.366
  02:18:20 STEP 1280 H= 320:00 +CPU= 71.722
  02:19:29 STEP 1281 H= 320:15 +CPU= 68.780
  02:20:35 STEP 1282 H= 320:30 +CPU= 65.130
  02:21:43 STEP 1283 H= 320:45 +CPU= 66.630
  02:22:56 STEP 1284 H= 321:00 +CPU= 72.588
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001284
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001284
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001284
  02:23:59 STEP 1285 H= 321:15 +CPU= 61.343
  02:25:02 STEP 1286 H= 321:30 +CPU= 61.998
  02:26:09 STEP 1287 H= 321:45 +CPU= 66.554
  02:27:10 STEP 1288 H= 322:00 +CPU= 61.474
  02:28:07 STEP 1289 H= 322:15 +CPU= 55.385
  02:29:06 STEP 1290 H= 322:30 +CPU= 57.980
  02:30:04 STEP 1291 H= 322:45 +CPU= 57.432
  02:31:07 STEP 1292 H= 323:00 +CPU= 62.375
  02:32:07 STEP 1293 H= 323:15 +CPU= 59.274
  02:33:04 STEP 1294 H= 323:30 +CPU= 56.802
  02:34:10 STEP 1295 H= 323:45 +CPU= 64.280
  02:35:27 STEP 1296 H= 324:00 +CPU= 76.229
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001296
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001296
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001296
  02:36:24 STEP 1297 H= 324:15 +CPU= 55.888
  02:37:23 STEP 1298 H= 324:30 +CPU= 57.776
  02:38:15 STEP 1299 H= 324:45 +CPU= 51.726
  02:39:13 STEP 1300 H= 325:00 +CPU= 57.983
  02:40:05 STEP 1301 H= 325:15 +CPU= 51.308
  02:40:59 STEP 1302 H= 325:30 +CPU= 53.562
  02:41:55 STEP 1303 H= 325:45 +CPU= 55.362
  02:42:51 STEP 1304 H= 326:00 +CPU= 55.669
  02:43:44 STEP 1305 H= 326:15 +CPU= 52.129
  02:44:38 STEP 1306 H= 326:30 +CPU= 53.142
  02:45:38 STEP 1307 H= 326:45 +CPU= 59.069
  02:46:49 STEP 1308 H= 327:00 +CPU= 69.460
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001308
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001308
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001308
  02:47:41 STEP 1309 H= 327:15 +CPU= 51.012
  02:48:35 STEP 1310 H= 327:30 +CPU= 52.790
  02:49:38 STEP 1311 H= 327:45 +CPU= 61.833
  02:50:34 STEP 1312 H= 328:00 +CPU= 54.867
  02:51:31 STEP 1313 H= 328:15 +CPU= 55.846
  02:52:27 STEP 1314 H= 328:30 +CPU= 55.330
  02:53:25 STEP 1315 H= 328:45 +CPU= 57.244
  02:54:23 STEP 1316 H= 329:00 +CPU= 57.725
  02:55:19 STEP 1317 H= 329:15 +CPU= 54.779
  02:56:15 STEP 1318 H= 329:30 +CPU= 55.196
  02:57:15 STEP 1319 H= 329:45 +CPU= 59.692
  02:58:26 STEP 1320 H= 330:00 +CPU= 70.527
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001320
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001320
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001320
  02:59:26 STEP 1321 H= 330:15 +CPU= 58.776
  03:00:25 STEP 1322 H= 330:30 +CPU= 58.452
  03:01:18 STEP 1323 H= 330:45 +CPU= 52.131
  03:02:20 STEP 1324 H= 331:00 +CPU= 60.206
  03:03:13 STEP 1325 H= 331:15 +CPU= 52.539
  03:04:12 STEP 1326 H= 331:30 +CPU= 58.972
  03:05:17 STEP 1327 H= 331:45 +CPU= 63.007
  03:06:15 STEP 1328 H= 332:00 +CPU= 57.842
  03:07:13 STEP 1329 H= 332:15 +CPU= 56.928
  03:08:05 STEP 1330 H= 332:30 +CPU= 51.576
  03:08:58 STEP 1331 H= 332:45 +CPU= 51.894
  03:10:03 STEP 1332 H= 333:00 +CPU= 64.702
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001332
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001332
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001332
  03:10:59 STEP 1333 H= 333:15 +CPU= 54.832
  03:11:53 STEP 1334 H= 333:30 +CPU= 53.892
  03:12:43 STEP 1335 H= 333:45 +CPU= 49.799
  03:13:50 STEP 1336 H= 334:00 +CPU= 65.376
  03:14:48 STEP 1337 H= 334:15 +CPU= 57.119
  03:15:47 STEP 1338 H= 334:30 +CPU= 58.249
  03:16:41 STEP 1339 H= 334:45 +CPU= 54.263
  03:17:44 STEP 1340 H= 335:00 +CPU= 62.190
  03:18:35 STEP 1341 H= 335:15 +CPU= 50.611
  03:19:27 STEP 1342 H= 335:30 +CPU= 51.712
  03:20:25 STEP 1343 H= 335:45 +CPU= 57.507
  03:21:32 STEP 1344 H= 336:00 +CPU= 66.949
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001344
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001344
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001248
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001260
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001272
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001284
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001296
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001308
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001320
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001332
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001332
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001332
Zipping up the intermediate file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_13.zip
Uploading the intermediate file: upload_file_13.zip
  03:22:29 STEP 1345 H= 336:15 +CPU= 55.731
  03:23:30 STEP 1346 H= 336:30 +CPU= 59.069
  03:24:23 STEP 1347 H= 336:45 +CPU= 51.639
  03:25:19 STEP 1348 H= 337:00 +CPU= 56.300
  03:26:22 STEP 1349 H= 337:15 +CPU= 61.909
  03:27:18 STEP 1350 H= 337:30 +CPU= 55.969
  03:28:11 STEP 1351 H= 337:45 +CPU= 52.790
  03:29:13 STEP 1352 H= 338:00 +CPU= 60.850
  03:30:10 STEP 1353 H= 338:15 +CPU= 56.628
  03:31:07 STEP 1354 H= 338:30 +CPU= 55.756
  03:32:08 STEP 1355 H= 338:45 +CPU= 59.730
  03:33:16 STEP 1356 H= 339:00 +CPU= 67.765
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001356
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001356
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001356
  03:34:17 STEP 1357 H= 339:15 +CPU= 60.672
  03:35:14 STEP 1358 H= 339:30 +CPU= 55.984
  03:36:11 STEP 1359 H= 339:45 +CPU= 57.075
  03:37:08 STEP 1360 H= 340:00 +CPU= 55.809
  03:38:11 STEP 1361 H= 340:15 +CPU= 62.385
  03:39:09 STEP 1362 H= 340:30 +CPU= 57.355
  03:40:02 STEP 1363 H= 340:45 +CPU= 52.220
  03:41:00 STEP 1364 H= 341:00 +CPU= 57.217
  03:41:57 STEP 1365 H= 341:15 +CPU= 56.441
  03:42:54 STEP 1366 H= 341:30 +CPU= 55.015
  03:43:48 STEP 1367 H= 341:45 +CPU= 53.301
  03:44:51 STEP 1368 H= 342:00 +CPU= 62.024
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001368
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001368
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001368
  03:45:48 STEP 1369 H= 342:15 +CPU= 55.759
  03:46:41 STEP 1370 H= 342:30 +CPU= 52.355
  03:47:48 STEP 1371 H= 342:45 +CPU= 65.807
  03:48:46 STEP 1372 H= 343:00 +CPU= 56.723
  03:49:44 STEP 1373 H= 343:15 +CPU= 57.338
  03:50:40 STEP 1374 H= 343:30 +CPU= 54.696
  03:51:36 STEP 1375 H= 343:45 +CPU= 56.347
  03:52:36 STEP 1376 H= 344:00 +CPU= 58.304
  03:53:38 STEP 1377 H= 344:15 +CPU= 60.249
  03:54:33 STEP 1378 H= 344:30 +CPU= 54.218
  03:55:29 STEP 1379 H= 344:45 +CPU= 53.790
  03:56:38 STEP 1380 H= 345:00 +CPU= 68.299
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001380
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001380
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001380
  03:57:33 STEP 1381 H= 345:15 +CPU= 53.323
  03:58:37 STEP 1382 H= 345:30 +CPU= 63.896
  03:59:34 STEP 1383 H= 345:45 +CPU= 55.725
  04:00:41 STEP 1384 H= 346:00 +CPU= 66.735
  04:01:33 STEP 1385 H= 346:15 +CPU= 51.546
  04:02:28 STEP 1386 H= 346:30 +CPU= 53.879
  04:03:29 STEP 1387 H= 346:45 +CPU= 60.365
  04:04:28 STEP 1388 H= 347:00 +CPU= 59.034
  04:05:26 STEP 1389 H= 347:15 +CPU= 57.170
  04:06:23 STEP 1390 H= 347:30 +CPU= 56.720
  04:07:13 STEP 1391 H= 347:45 +CPU= 49.396
  04:08:36 STEP 1392 H= 348:00 +CPU= 82.460
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001392
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001392
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001392
  04:09:33 STEP 1393 H= 348:15 +CPU= 55.869
  04:10:35 STEP 1394 H= 348:30 +CPU= 60.722
  04:11:34 STEP 1395 H= 348:45 +CPU= 58.582
  04:12:34 STEP 1396 H= 349:00 +CPU= 57.526
  04:13:38 STEP 1397 H= 349:15 +CPU= 63.867
  04:14:32 STEP 1398 H= 349:30 +CPU= 53.308
  04:15:35 STEP 1399 H= 349:45 +CPU= 61.330
  04:16:40 STEP 1400 H= 350:00 +CPU= 64.175
  04:17:43 STEP 1401 H= 350:15 +CPU= 62.449
  04:18:48 STEP 1402 H= 350:30 +CPU= 64.433
  04:19:53 STEP 1403 H= 350:45 +CPU= 63.970
  04:21:00 STEP 1404 H= 351:00 +CPU= 66.185
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001404
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001404
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001404
  04:21:54 STEP 1405 H= 351:15 +CPU= 53.621
  04:22:52 STEP 1406 H= 351:30 +CPU= 57.027
  04:23:52 STEP 1407 H= 351:45 +CPU= 60.219
  04:24:55 STEP 1408 H= 352:00 +CPU= 62.128
  04:25:58 STEP 1409 H= 352:15 +CPU= 61.916
  04:26:57 STEP 1410 H= 352:30 +CPU= 58.222
  04:27:55 STEP 1411 H= 352:45 +CPU= 57.270
  04:28:56 STEP 1412 H= 353:00 +CPU= 61.061
  04:29:56 STEP 1413 H= 353:15 +CPU= 59.300
  04:30:52 STEP 1414 H= 353:30 +CPU= 55.802
  04:31:48 STEP 1415 H= 353:45 +CPU= 54.756
  04:33:00 STEP 1416 H= 354:00 +CPU= 71.134
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001416
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001416
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001416
  04:34:02 STEP 1417 H= 354:15 +CPU= 59.885
  04:34:57 STEP 1418 H= 354:30 +CPU= 54.830
  04:35:57 STEP 1419 H= 354:45 +CPU= 58.174
  04:36:53 STEP 1420 H= 355:00 +CPU= 56.378
  04:37:57 STEP 1421 H= 355:15 +CPU= 62.819
  04:38:50 STEP 1422 H= 355:30 +CPU= 51.696
  04:39:47 STEP 1423 H= 355:45 +CPU= 56.471
  04:40:48 STEP 1424 H= 356:00 +CPU= 60.522
  04:41:51 STEP 1425 H= 356:15 +CPU= 61.864
  04:42:44 STEP 1426 H= 356:30 +CPU= 52.045
  04:43:39 STEP 1427 H= 356:45 +CPU= 54.381
  04:44:51 STEP 1428 H= 357:00 +CPU= 70.038
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001428
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001428
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001428
  04:45:54 STEP 1429 H= 357:15 +CPU= 62.604
  04:46:51 STEP 1430 H= 357:30 +CPU= 56.286
  04:47:46 STEP 1431 H= 357:45 +CPU= 54.017
  04:48:31 STEP 1432 H= 358:00 +CPU= 44.709
  04:49:22 STEP 1433 H= 358:15 +CPU= 50.739
  04:50:14 STEP 1434 H= 358:30 +CPU= 51.718
  04:51:12 STEP 1435 H= 358:45 +CPU= 57.933
  04:52:13 STEP 1436 H= 359:00 +CPU= 60.011
  04:53:10 STEP 1437 H= 359:15 +CPU= 56.353
  04:54:08 STEP 1438 H= 359:30 +CPU= 57.628
  04:55:08 STEP 1439 H= 359:45 +CPU= 58.823
  04:56:15 STEP 1440 H= 360:00 +CPU= 66.486
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.2054066226E-21      I       0.5270871945E-06      I
ICloud fractionI      0.0000000000E+00      I       0.1392062135E-01      I
I  Relat. Hum. I      0.5089469768E-01      I       0.1578182889E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1673173015E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    299385748 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS427693.9 Mbytes/s, TOTAL TIME=    0.0131( 5.3%)
 04:56:15 STEP 1440 H= 360:00 +CPU= 66.486
 END CNT3
 NSTEP =  1440 CNT0   000000000  51.697  51.697  52.201   0.990               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        22880/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    403    0       46      1044068    22880 ************    93.8     2.4    91.3     4.4   180.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.98      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  833   12042.1   12042.1     21.79      0.00
   9 SPCM     - SPECTRAL COMP.                     768    3015.4    3015.4      5.03      0.00
  10 SCAN2M - PHYSICS                              769   24795.1   24795.1     41.43      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 64    2238.2    2238.2      0.31      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 33     147.6     147.6      0.01      0.00
  17 GRIDFPOS IN CNT4                               64      65.8      65.8      0.01      0.00
  27 RESTART FILES - WRITING                        16    8952.9    8952.9      0.31      0.00
  28 RESTART FILES - READING                         1    6607.1    6607.1      0.01      0.00
  29 SU4FPOS IN CNT4                                64       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                64   10618.1   10618.1      1.48      0.00
  31 POSDDH IN STEPO                                 4       9.6       9.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    769   11053.9   11053.9     18.47      0.00
  39 SU0YOMB                                         1     680.6     680.6      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    769      68.9      68.9      0.12      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              768    1629.3    1629.3      2.72      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              768     154.3     154.3      0.26      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 33       1.2       1.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     1668     772.3     772.3      2.80      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      1600     394.0     394.0      1.37      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       1600     179.2     179.2      0.62      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      1668     265.6     265.6      0.96      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      78.7      78.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           1668     178.0     178.0      0.65      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           1600     113.9     113.9      0.40      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           1668     284.0     284.0      1.03      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           1600     138.1     138.1      0.48      0.00
 400 GSTATS                                      89902       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 83594       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    46026.5 CPU TIME   45491.7 VECTOR TIME    45491.7


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       28.2
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/test/slots/32/ICMGGh7zg+001440
Moving to projects directory: /home/test/slots/32/ICMSHh7zg+001440
Moving to projects directory: /home/test/slots/32/ICMUAh7zg+001440
Adding to the zip: /home/test/slots/32/NODE.001_01
Adding to the zip: /home/test/slots/32/ifs.stat
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001344
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001356
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001380
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001416
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001392
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMGGh7zg+001404
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001428
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001368
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMUAh7zg+001440
Adding to the zip: /home/test/projects/climateprediction.net/oifs_43r3_bl_12282158/ICMSHh7zg+001344
Zipping up the final file: /home/test/projects/climateprediction.net/oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
04:59:48 (8436): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a097_2016092300_20_1017_12282158_0_r216698592_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 10:08:48 1545820 22433725 oifs_43r3_bl_a097_2016092300_20_1017_12282158_0 1,295,100 80,350 0.0620


©2024 cpdn.org