climateprediction.net home page
Task 22435090

Task 22435090

Name oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0
Workunit 12283511
Created 7 Jun 2024, 13:17:04 UTC
Sent 8 Jun 2024, 2:25:16 UTC
Report deadline 7 Aug 2024, 2:25:16 UTC
Received 9 Jun 2024, 0:54:26 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1536520
Run time 21 hours 58 min 53 sec
CPU time 21 hours 32 min 18 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.63 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,341.09 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
directory: /home/steve/slots/7/ICMGGh7zg+000780
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000780
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000780
  09:47:19 STEP  781 H= 195:15 +CPU= 54.839
  09:48:16 STEP  782 H= 195:30 +CPU= 54.424
  09:49:14 STEP  783 H= 195:45 +CPU= 56.891
  09:50:11 STEP  784 H= 196:00 +CPU= 56.416
  09:51:10 STEP  785 H= 196:15 +CPU= 58.295
  09:52:07 STEP  786 H= 196:30 +CPU= 56.478
  09:53:05 STEP  787 H= 196:45 +CPU= 57.835
  09:54:05 STEP  788 H= 197:00 +CPU= 59.407
  09:55:04 STEP  789 H= 197:15 +CPU= 57.692
  09:56:01 STEP  790 H= 197:30 +CPU= 56.441
  09:56:58 STEP  791 H= 197:45 +CPU= 56.327
  09:58:06 STEP  792 H= 198:00 +CPU= 68.042
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000792
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000792
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000792
  09:59:01 STEP  793 H= 198:15 +CPU= 54.585
  09:59:57 STEP  794 H= 198:30 +CPU= 55.482
  10:00:53 STEP  795 H= 198:45 +CPU= 55.228
  10:01:49 STEP  796 H= 199:00 +CPU= 55.772
  10:02:46 STEP  797 H= 199:15 +CPU= 55.498
  10:03:43 STEP  798 H= 199:30 +CPU= 56.659
  10:04:39 STEP  799 H= 199:45 +CPU= 56.175
  10:05:37 STEP  800 H= 200:00 +CPU= 57.258
  10:06:33 STEP  801 H= 200:15 +CPU= 55.186
  10:07:30 STEP  802 H= 200:30 +CPU= 56.210
  10:08:27 STEP  803 H= 200:45 +CPU= 56.599
  10:09:35 STEP  804 H= 201:00 +CPU= 66.999
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000804
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000804
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000804
  10:10:31 STEP  805 H= 201:15 +CPU= 55.649
  10:11:26 STEP  806 H= 201:30 +CPU= 54.862
  10:12:23 STEP  807 H= 201:45 +CPU= 56.263
  10:13:20 STEP  808 H= 202:00 +CPU= 56.023
  10:14:15 STEP  809 H= 202:15 +CPU= 55.417
  10:15:12 STEP  810 H= 202:30 +CPU= 56.156
  10:16:08 STEP  811 H= 202:45 +CPU= 55.134
  10:17:05 STEP  812 H= 203:00 +CPU= 57.135
  10:18:01 STEP  813 H= 203:15 +CPU= 54.787
  10:18:56 STEP  814 H= 203:30 +CPU= 54.515
  10:19:52 STEP  815 H= 203:45 +CPU= 56.023
  10:21:05 STEP  816 H= 204:00 +CPU= 71.709
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000816
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000816
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000816
  10:22:02 STEP  817 H= 204:15 +CPU= 56.326
  10:22:58 STEP  818 H= 204:30 +CPU= 55.300
  10:23:54 STEP  819 H= 204:45 +CPU= 55.670
  10:24:51 STEP  820 H= 205:00 +CPU= 56.371
  10:25:46 STEP  821 H= 205:15 +CPU= 54.630
  10:26:42 STEP  822 H= 205:30 +CPU= 55.884
  10:27:39 STEP  823 H= 205:45 +CPU= 55.564
  10:28:35 STEP  824 H= 206:00 +CPU= 55.637
  10:29:31 STEP  825 H= 206:15 +CPU= 55.524
  10:30:28 STEP  826 H= 206:30 +CPU= 56.615
  10:31:25 STEP  827 H= 206:45 +CPU= 57.038
  10:32:36 STEP  828 H= 207:00 +CPU= 69.341
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000828
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000828
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000828
  10:33:33 STEP  829 H= 207:15 +CPU= 57.125
  10:34:30 STEP  830 H= 207:30 +CPU= 56.648
  10:35:28 STEP  831 H= 207:45 +CPU= 56.844
  10:36:27 STEP  832 H= 208:00 +CPU= 58.559
  10:37:25 STEP  833 H= 208:15 +CPU= 57.481
  10:38:23 STEP  834 H= 208:30 +CPU= 57.772
  10:39:20 STEP  835 H= 208:45 +CPU= 56.500
  10:40:18 STEP  836 H= 209:00 +CPU= 57.230
  10:41:15 STEP  837 H= 209:15 +CPU= 56.284
  10:42:12 STEP  838 H= 209:30 +CPU= 56.008
  10:43:09 STEP  839 H= 209:45 +CPU= 56.476
  10:44:18 STEP  840 H= 210:00 +CPU= 68.963
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000840
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000840
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000840
  10:45:15 STEP  841 H= 210:15 +CPU= 56.307
  10:46:12 STEP  842 H= 210:30 +CPU= 56.130
  10:47:11 STEP  843 H= 210:45 +CPU= 57.577
  10:48:09 STEP  844 H= 211:00 +CPU= 56.746
  10:49:06 STEP  845 H= 211:15 +CPU= 56.909
  10:50:04 STEP  846 H= 211:30 +CPU= 57.662
  10:51:01 STEP  847 H= 211:45 +CPU= 56.385
  10:52:00 STEP  848 H= 212:00 +CPU= 58.120
  10:52:56 STEP  849 H= 212:15 +CPU= 55.424
  10:53:53 STEP  850 H= 212:30 +CPU= 56.622
  10:54:49 STEP  851 H= 212:45 +CPU= 55.977
  10:56:00 STEP  852 H= 213:00 +CPU= 68.120
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000852
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000852
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000852
  10:57:00 STEP  853 H= 213:15 +CPU= 56.117
  10:57:59 STEP  854 H= 213:30 +CPU= 56.248
  10:58:56 STEP  855 H= 213:45 +CPU= 54.974
  10:59:55 STEP  856 H= 214:00 +CPU= 56.487
  11:00:53 STEP  857 H= 214:15 +CPU= 56.527
  11:01:53 STEP  858 H= 214:30 +CPU= 54.935
  11:02:56 STEP  859 H= 214:45 +CPU= 54.260
  11:03:59 STEP  860 H= 215:00 +CPU= 54.236
  11:05:02 STEP  861 H= 215:15 +CPU= 56.030
  11:06:05 STEP  862 H= 215:30 +CPU= 55.236
  11:07:09 STEP  863 H= 215:45 +CPU= 58.522
  11:08:31 STEP  864 H= 216:00 +CPU= 74.432
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000864
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000864
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_8.zip
Uploading the intermediate file: upload_file_8.zip
  11:09:34 STEP  865 H= 216:15 +CPU= 57.425
  11:10:40 STEP  866 H= 216:30 +CPU= 56.986
  11:11:46 STEP  867 H= 216:45 +CPU= 57.963
  11:12:56 STEP  868 H= 217:00 +CPU= 59.669
  11:14:02 STEP  869 H= 217:15 +CPU= 59.394
  11:15:11 STEP  870 H= 217:30 +CPU= 59.671
  11:16:17 STEP  871 H= 217:45 +CPU= 56.690
  11:17:23 STEP  872 H= 218:00 +CPU= 58.525
  11:18:25 STEP  873 H= 218:15 +CPU= 58.772
  11:19:27 STEP  874 H= 218:30 +CPU= 58.227
  11:20:30 STEP  875 H= 218:45 +CPU= 58.848
  11:21:49 STEP  876 H= 219:00 +CPU= 70.887
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000876
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000876
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000876
  11:22:53 STEP  877 H= 219:15 +CPU= 58.484
  11:23:56 STEP  878 H= 219:30 +CPU= 57.476
  11:24:57 STEP  879 H= 219:45 +CPU= 57.680
  11:26:02 STEP  880 H= 220:00 +CPU= 57.402
  11:27:08 STEP  881 H= 220:15 +CPU= 58.879
  11:28:16 STEP  882 H= 220:30 +CPU= 60.045
  11:29:21 STEP  883 H= 220:45 +CPU= 58.549
  11:30:26 STEP  884 H= 221:00 +CPU= 58.503
  11:31:31 STEP  885 H= 221:15 +CPU= 57.135
  11:32:37 STEP  886 H= 221:30 +CPU= 58.554
  11:33:44 STEP  887 H= 221:45 +CPU= 57.567
  11:35:03 STEP  888 H= 222:00 +CPU= 70.291
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000888
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000888
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000888
  11:36:09 STEP  889 H= 222:15 +CPU= 58.109
  11:37:14 STEP  890 H= 222:30 +CPU= 57.554
  11:38:18 STEP  891 H= 222:45 +CPU= 58.338
  11:39:23 STEP  892 H= 223:00 +CPU= 59.136
  11:40:29 STEP  893 H= 223:15 +CPU= 57.723
  11:41:33 STEP  894 H= 223:30 +CPU= 56.948
  11:42:34 STEP  895 H= 223:45 +CPU= 56.406
  11:43:37 STEP  896 H= 224:00 +CPU= 58.431
  11:44:39 STEP  897 H= 224:15 +CPU= 57.850
  11:45:41 STEP  898 H= 224:30 +CPU= 56.226
  11:46:40 STEP  899 H= 224:45 +CPU= 56.473
  11:47:50 STEP  900 H= 225:00 +CPU= 68.846
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000900
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000900
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000900
  11:48:48 STEP  901 H= 225:15 +CPU= 57.339
  11:49:45 STEP  902 H= 225:30 +CPU= 56.564
  11:50:43 STEP  903 H= 225:45 +CPU= 57.757
  11:51:42 STEP  904 H= 226:00 +CPU= 58.198
  11:52:38 STEP  905 H= 226:15 +CPU= 55.796
  11:53:35 STEP  906 H= 226:30 +CPU= 56.697
  11:54:33 STEP  907 H= 226:45 +CPU= 56.823
  11:55:32 STEP  908 H= 227:00 +CPU= 59.078
  11:56:30 STEP  909 H= 227:15 +CPU= 57.400
  11:57:26 STEP  910 H= 227:30 +CPU= 55.600
  11:58:25 STEP  911 H= 227:45 +CPU= 57.789
  11:59:40 STEP  912 H= 228:00 +CPU= 74.869
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000912
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000912
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000912
  12:00:36 STEP  913 H= 228:15 +CPU= 55.303
  12:01:32 STEP  914 H= 228:30 +CPU= 55.507
  12:02:29 STEP  915 H= 228:45 +CPU= 56.064
  12:03:25 STEP  916 H= 229:00 +CPU= 55.971
  12:04:21 STEP  917 H= 229:15 +CPU= 55.208
  12:05:17 STEP  918 H= 229:30 +CPU= 55.099
  12:06:14 STEP  919 H= 229:45 +CPU= 56.645
  12:07:11 STEP  920 H= 230:00 +CPU= 56.721
  12:08:08 STEP  921 H= 230:15 +CPU= 56.151
  12:09:06 STEP  922 H= 230:30 +CPU= 57.302
  12:10:03 STEP  923 H= 230:45 +CPU= 56.158
  12:11:09 STEP  924 H= 231:00 +CPU= 66.017
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000924
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000924
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000924
  12:12:05 STEP  925 H= 231:15 +CPU= 55.053
  12:13:00 STEP  926 H= 231:30 +CPU= 54.844
  12:13:55 STEP  927 H= 231:45 +CPU= 54.372
  12:14:51 STEP  928 H= 232:00 +CPU= 55.731
  12:15:47 STEP  929 H= 232:15 +CPU= 55.214
  12:16:41 STEP  930 H= 232:30 +CPU= 54.022
  12:17:38 STEP  931 H= 232:45 +CPU= 56.116
  12:18:36 STEP  932 H= 233:00 +CPU= 57.705
  12:19:33 STEP  933 H= 233:15 +CPU= 56.362
  12:20:30 STEP  934 H= 233:30 +CPU= 56.334
  12:21:26 STEP  935 H= 233:45 +CPU= 55.478
  12:22:35 STEP  936 H= 234:00 +CPU= 66.419
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000936
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000936
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000936
  12:23:31 STEP  937 H= 234:15 +CPU= 55.418
  12:24:27 STEP  938 H= 234:30 +CPU= 55.292
  12:25:23 STEP  939 H= 234:45 +CPU= 56.273
  12:26:22 STEP  940 H= 235:00 +CPU= 58.611
  12:27:21 STEP  941 H= 235:15 +CPU= 57.555
  12:28:20 STEP  942 H= 235:30 +CPU= 58.423
  12:29:18 STEP  943 H= 235:45 +CPU= 57.928
  12:30:16 STEP  944 H= 236:00 +CPU= 57.525
  12:31:13 STEP  945 H= 236:15 +CPU= 56.279
  12:32:11 STEP  946 H= 236:30 +CPU= 57.257
  12:33:10 STEP  947 H= 236:45 +CPU= 58.882
  12:34:23 STEP  948 H= 237:00 +CPU= 72.150
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000948
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000948
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000948
  12:35:20 STEP  949 H= 237:15 +CPU= 56.482
  12:36:18 STEP  950 H= 237:30 +CPU= 56.966
  12:37:16 STEP  951 H= 237:45 +CPU= 57.716
  12:38:15 STEP  952 H= 238:00 +CPU= 57.987
  12:39:11 STEP  953 H= 238:15 +CPU= 55.729
  12:40:08 STEP  954 H= 238:30 +CPU= 56.224
  12:41:06 STEP  955 H= 238:45 +CPU= 56.675
  12:42:04 STEP  956 H= 239:00 +CPU= 57.491
  12:43:01 STEP  957 H= 239:15 +CPU= 56.583
  12:44:00 STEP  958 H= 239:30 +CPU= 58.035
  12:44:57 STEP  959 H= 239:45 +CPU= 56.779
  12:46:12 STEP  960 H= 240:00 +CPU= 74.440
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000960
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000960
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  12:47:10 STEP  961 H= 240:15 +CPU= 56.250
  12:48:10 STEP  962 H= 240:30 +CPU= 59.809
  12:49:07 STEP  963 H= 240:45 +CPU= 56.042
  12:50:07 STEP  964 H= 241:00 +CPU= 58.287
  12:51:01 STEP  965 H= 241:15 +CPU= 53.429
  12:51:53 STEP  966 H= 241:30 +CPU= 51.802
  12:52:46 STEP  967 H= 241:45 +CPU= 52.639
  12:53:39 STEP  968 H= 242:00 +CPU= 52.408
  12:54:16 STEP  969 H= 242:15 +CPU= 37.109
  12:54:55 STEP  970 H= 242:30 +CPU= 38.827
  12:55:34 STEP  971 H= 242:45 +CPU= 38.438
  12:56:21 STEP  972 H= 243:00 +CPU= 46.822
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000972
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000972
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000972
  12:56:59 STEP  973 H= 243:15 +CPU= 37.355
  12:57:42 STEP  974 H= 243:30 +CPU= 42.820
  12:58:29 STEP  975 H= 243:45 +CPU= 46.651
  12:59:16 STEP  976 H= 244:00 +CPU= 46.896
  13:00:05 STEP  977 H= 244:15 +CPU= 48.244
  13:00:51 STEP  978 H= 244:30 +CPU= 46.245
  13:01:39 STEP  979 H= 244:45 +CPU= 47.329
  13:02:28 STEP  980 H= 245:00 +CPU= 48.073
  13:03:15 STEP  981 H= 245:15 +CPU= 47.210
  13:04:03 STEP  982 H= 245:30 +CPU= 47.098
  13:04:50 STEP  983 H= 245:45 +CPU= 46.833
  13:05:48 STEP  984 H= 246:00 +CPU= 57.764
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000984
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000984
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000984
  13:06:36 STEP  985 H= 246:15 +CPU= 46.851
  13:07:22 STEP  986 H= 246:30 +CPU= 46.081
  13:08:11 STEP  987 H= 246:45 +CPU= 48.367
  13:08:59 STEP  988 H= 247:00 +CPU= 47.861
  13:09:47 STEP  989 H= 247:15 +CPU= 47.802
  13:10:35 STEP  990 H= 247:30 +CPU= 47.933
  13:11:24 STEP  991 H= 247:45 +CPU= 48.577
  13:12:13 STEP  992 H= 248:00 +CPU= 48.623
  13:13:01 STEP  993 H= 248:15 +CPU= 47.592
  13:13:49 STEP  994 H= 248:30 +CPU= 47.687
  13:14:37 STEP  995 H= 248:45 +CPU= 47.429
  13:15:36 STEP  996 H= 249:00 +CPU= 58.444
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+000996
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+000996
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+000996
  13:16:23 STEP  997 H= 249:15 +CPU= 46.815
  13:17:11 STEP  998 H= 249:30 +CPU= 47.130
  13:17:58 STEP  999 H= 249:45 +CPU= 46.847
  13:18:50 STEP 1000 H= 250:00 +CPU= 48.896
  13:19:39 STEP 1001 H= 250:15 +CPU= 46.898
  13:20:27 STEP 1002 H= 250:30 +CPU= 46.982
  13:21:16 STEP 1003 H= 250:45 +CPU= 47.472
  13:22:06 STEP 1004 H= 251:00 +CPU= 47.629
  13:22:56 STEP 1005 H= 251:15 +CPU= 46.679
  13:23:49 STEP 1006 H= 251:30 +CPU= 46.092
  13:24:40 STEP 1007 H= 251:45 +CPU= 46.079
  13:25:48 STEP 1008 H= 252:00 +CPU= 62.761
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001008
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001008
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001008
  13:26:38 STEP 1009 H= 252:15 +CPU= 46.971
  13:27:28 STEP 1010 H= 252:30 +CPU= 48.284
  13:28:17 STEP 1011 H= 252:45 +CPU= 47.180
  13:29:08 STEP 1012 H= 253:00 +CPU= 48.176
  13:29:55 STEP 1013 H= 253:15 +CPU= 46.522
  13:30:43 STEP 1014 H= 253:30 +CPU= 46.856
  13:31:30 STEP 1015 H= 253:45 +CPU= 47.213
  13:32:19 STEP 1016 H= 254:00 +CPU= 48.376
  13:33:08 STEP 1017 H= 254:15 +CPU= 48.044
  13:33:56 STEP 1018 H= 254:30 +CPU= 47.411
  13:34:45 STEP 1019 H= 254:45 +CPU= 47.222
  13:35:48 STEP 1020 H= 255:00 +CPU= 58.208
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001020
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001020
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001020
  13:36:37 STEP 1021 H= 255:15 +CPU= 46.252
  13:37:26 STEP 1022 H= 255:30 +CPU= 47.112
  13:38:16 STEP 1023 H= 255:45 +CPU= 47.016
  13:39:05 STEP 1024 H= 256:00 +CPU= 48.333
  13:39:54 STEP 1025 H= 256:15 +CPU= 47.035
  13:40:42 STEP 1026 H= 256:30 +CPU= 46.877
  13:41:30 STEP 1027 H= 256:45 +CPU= 47.075
  13:42:19 STEP 1028 H= 257:00 +CPU= 47.556
  13:43:08 STEP 1029 H= 257:15 +CPU= 46.732
  13:43:58 STEP 1030 H= 257:30 +CPU= 47.220
  13:44:46 STEP 1031 H= 257:45 +CPU= 47.039
  13:45:51 STEP 1032 H= 258:00 +CPU= 59.487
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001032
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001032
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001032
  13:46:43 STEP 1033 H= 258:15 +CPU= 47.783
  13:47:33 STEP 1034 H= 258:30 +CPU= 48.117
  13:48:23 STEP 1035 H= 258:45 +CPU= 48.088
  13:49:16 STEP 1036 H= 259:00 +CPU= 48.893
  13:50:06 STEP 1037 H= 259:15 +CPU= 47.587
  13:50:56 STEP 1038 H= 259:30 +CPU= 46.887
  13:51:44 STEP 1039 H= 259:45 +CPU= 47.194
  13:52:33 STEP 1040 H= 260:00 +CPU= 48.778
  13:53:20 STEP 1041 H= 260:15 +CPU= 46.819
  13:54:07 STEP 1042 H= 260:30 +CPU= 46.886
  13:54:55 STEP 1043 H= 260:45 +CPU= 47.640
  13:55:53 STEP 1044 H= 261:00 +CPU= 56.823
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001044
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001044
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001044
  13:56:40 STEP 1045 H= 261:15 +CPU= 47.264
  13:57:28 STEP 1046 H= 261:30 +CPU= 47.083
  13:58:15 STEP 1047 H= 261:45 +CPU= 46.881
  13:59:05 STEP 1048 H= 262:00 +CPU= 48.887
  13:59:51 STEP 1049 H= 262:15 +CPU= 46.368
  14:00:39 STEP 1050 H= 262:30 +CPU= 47.545
  14:01:27 STEP 1051 H= 262:45 +CPU= 46.960
  14:02:16 STEP 1052 H= 263:00 +CPU= 48.513
  14:03:03 STEP 1053 H= 263:15 +CPU= 47.062
  14:03:51 STEP 1054 H= 263:30 +CPU= 47.626
  14:04:39 STEP 1055 H= 263:45 +CPU= 47.592
  14:05:44 STEP 1056 H= 264:00 +CPU= 64.527
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001056
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001056
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_10.zip
Uploading the intermediate file: upload_file_10.zip
  14:06:33 STEP 1057 H= 264:15 +CPU= 47.745
  14:07:23 STEP 1058 H= 264:30 +CPU= 48.999
  14:08:11 STEP 1059 H= 264:45 +CPU= 48.359
  14:09:00 STEP 1060 H= 265:00 +CPU= 48.121
  14:09:48 STEP 1061 H= 265:15 +CPU= 46.090
  14:10:37 STEP 1062 H= 265:30 +CPU= 48.811
  14:11:25 STEP 1063 H= 265:45 +CPU= 47.622
  14:12:14 STEP 1064 H= 266:00 +CPU= 48.677
  14:13:01 STEP 1065 H= 266:15 +CPU= 47.081
  14:13:49 STEP 1066 H= 266:30 +CPU= 46.930
  14:14:36 STEP 1067 H= 266:45 +CPU= 46.640
  14:15:35 STEP 1068 H= 267:00 +CPU= 58.982
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001068
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001068
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001068
  14:16:22 STEP 1069 H= 267:15 +CPU= 46.305
  14:17:10 STEP 1070 H= 267:30 +CPU= 47.589
  14:17:57 STEP 1071 H= 267:45 +CPU= 46.653
  14:18:46 STEP 1072 H= 268:00 +CPU= 48.468
  14:19:34 STEP 1073 H= 268:15 +CPU= 47.458
  14:20:21 STEP 1074 H= 268:30 +CPU= 47.381
  14:21:09 STEP 1075 H= 268:45 +CPU= 47.509
  14:21:58 STEP 1076 H= 269:00 +CPU= 48.505
  14:22:46 STEP 1077 H= 269:15 +CPU= 47.797
  14:23:35 STEP 1078 H= 269:30 +CPU= 47.205
  14:24:25 STEP 1079 H= 269:45 +CPU= 47.262
  14:25:27 STEP 1080 H= 270:00 +CPU= 58.401
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001080
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001080
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001080
  14:26:15 STEP 1081 H= 270:15 +CPU= 47.018
  14:27:05 STEP 1082 H= 270:30 +CPU= 48.701
  14:27:52 STEP 1083 H= 270:45 +CPU= 46.879
  14:28:41 STEP 1084 H= 271:00 +CPU= 48.293
  14:29:29 STEP 1085 H= 271:15 +CPU= 47.488
  14:30:16 STEP 1086 H= 271:30 +CPU= 46.789
  14:31:04 STEP 1087 H= 271:45 +CPU= 47.666
  14:31:53 STEP 1088 H= 272:00 +CPU= 48.343
  14:32:40 STEP 1089 H= 272:15 +CPU= 46.981
  14:33:28 STEP 1090 H= 272:30 +CPU= 47.427
  14:34:17 STEP 1091 H= 272:45 +CPU= 48.046
  14:35:15 STEP 1092 H= 273:00 +CPU= 57.652
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001092
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001092
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001092
  14:36:02 STEP 1093 H= 273:15 +CPU= 47.059
  14:36:50 STEP 1094 H= 273:30 +CPU= 47.447
  14:37:38 STEP 1095 H= 273:45 +CPU= 47.434
  14:38:27 STEP 1096 H= 274:00 +CPU= 48.889
  14:39:15 STEP 1097 H= 274:15 +CPU= 47.578
  14:40:05 STEP 1098 H= 274:30 +CPU= 48.383
  14:40:54 STEP 1099 H= 274:45 +CPU= 47.669
  14:41:43 STEP 1100 H= 275:00 +CPU= 47.680
  14:42:32 STEP 1101 H= 275:15 +CPU= 47.910
  14:43:20 STEP 1102 H= 275:30 +CPU= 47.849
  14:44:09 STEP 1103 H= 275:45 +CPU= 48.339
  14:45:13 STEP 1104 H= 276:00 +CPU= 63.039
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001104
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001104
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001104
  14:46:00 STEP 1105 H= 276:15 +CPU= 46.646
  14:46:47 STEP 1106 H= 276:30 +CPU= 46.481
  14:47:33 STEP 1107 H= 276:45 +CPU= 46.246
  14:48:21 STEP 1108 H= 277:00 +CPU= 47.466
  14:49:09 STEP 1109 H= 277:15 +CPU= 46.952
  14:49:56 STEP 1110 H= 277:30 +CPU= 46.809
  14:50:42 STEP 1111 H= 277:45 +CPU= 45.703
  14:51:30 STEP 1112 H= 278:00 +CPU= 47.685
  14:52:18 STEP 1113 H= 278:15 +CPU= 47.130
  14:53:05 STEP 1114 H= 278:30 +CPU= 46.846
  14:53:53 STEP 1115 H= 278:45 +CPU= 47.701
  14:54:52 STEP 1116 H= 279:00 +CPU= 58.807
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001116
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001116
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001116
  14:55:40 STEP 1117 H= 279:15 +CPU= 47.317
  14:56:29 STEP 1118 H= 279:30 +CPU= 48.053
  14:57:17 STEP 1119 H= 279:45 +CPU= 46.769
  14:58:06 STEP 1120 H= 280:00 +CPU= 48.784
  14:58:56 STEP 1121 H= 280:15 +CPU= 49.161
  14:59:44 STEP 1122 H= 280:30 +CPU= 47.539
  15:00:31 STEP 1123 H= 280:45 +CPU= 47.293
  15:01:20 STEP 1124 H= 281:00 +CPU= 48.085
  15:02:08 STEP 1125 H= 281:15 +CPU= 47.827
  15:02:56 STEP 1126 H= 281:30 +CPU= 47.742
  15:03:44 STEP 1127 H= 281:45 +CPU= 47.299
  15:04:42 STEP 1128 H= 282:00 +CPU= 57.626
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001128
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001128
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001128
  15:05:29 STEP 1129 H= 282:15 +CPU= 46.307
  15:06:17 STEP 1130 H= 282:30 +CPU= 47.237
  15:07:04 STEP 1131 H= 282:45 +CPU= 47.025
  15:07:52 STEP 1132 H= 283:00 +CPU= 47.100
  15:08:40 STEP 1133 H= 283:15 +CPU= 47.659
  15:09:27 STEP 1134 H= 283:30 +CPU= 46.458
  15:10:15 STEP 1135 H= 283:45 +CPU= 47.828
  15:11:02 STEP 1136 H= 284:00 +CPU= 47.189
  15:11:50 STEP 1137 H= 284:15 +CPU= 47.236
  15:12:37 STEP 1138 H= 284:30 +CPU= 46.628
  15:13:25 STEP 1139 H= 284:45 +CPU= 47.538
  15:14:24 STEP 1140 H= 285:00 +CPU= 58.013
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001140
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001140
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001140
  15:15:13 STEP 1141 H= 285:15 +CPU= 48.214
  15:16:00 STEP 1142 H= 285:30 +CPU= 47.357
  15:16:47 STEP 1143 H= 285:45 +CPU= 46.653
  15:17:37 STEP 1144 H= 286:00 +CPU= 48.822
  15:18:25 STEP 1145 H= 286:15 +CPU= 47.900
  15:19:14 STEP 1146 H= 286:30 +CPU= 48.256
  15:20:02 STEP 1147 H= 286:45 +CPU= 47.848
  15:20:51 STEP 1148 H= 287:00 +CPU= 48.489
  15:21:39 STEP 1149 H= 287:15 +CPU= 47.346
  15:22:27 STEP 1150 H= 287:30 +CPU= 47.941
  15:23:14 STEP 1151 H= 287:45 +CPU= 46.811
  15:24:19 STEP 1152 H= 288:00 +CPU= 64.155
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001152
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001152
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_11.zip
Uploading the intermediate file: upload_file_11.zip
  15:25:08 STEP 1153 H= 288:15 +CPU= 47.742
  15:25:56 STEP 1154 H= 288:30 +CPU= 48.138
  15:26:45 STEP 1155 H= 288:45 +CPU= 48.079
  15:27:35 STEP 1156 H= 289:00 +CPU= 49.456
  15:28:24 STEP 1157 H= 289:15 +CPU= 47.842
  15:29:11 STEP 1158 H= 289:30 +CPU= 46.623
  15:30:00 STEP 1159 H= 289:45 +CPU= 48.761
  15:30:49 STEP 1160 H= 290:00 +CPU= 48.310
  15:31:37 STEP 1161 H= 290:15 +CPU= 47.670
  15:32:26 STEP 1162 H= 290:30 +CPU= 48.779
  15:33:14 STEP 1163 H= 290:45 +CPU= 47.560
  15:34:13 STEP 1164 H= 291:00 +CPU= 57.983
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001164
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001164
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001164
  15:35:01 STEP 1165 H= 291:15 +CPU= 47.810
  15:35:48 STEP 1166 H= 291:30 +CPU= 46.990
  15:36:36 STEP 1167 H= 291:45 +CPU= 47.692
  15:37:24 STEP 1168 H= 292:00 +CPU= 47.097
  15:38:12 STEP 1169 H= 292:15 +CPU= 48.217
  15:39:00 STEP 1170 H= 292:30 +CPU= 47.417
  15:39:48 STEP 1171 H= 292:45 +CPU= 47.390
  15:40:37 STEP 1172 H= 293:00 +CPU= 48.969
  15:41:25 STEP 1173 H= 293:15 +CPU= 47.635
  15:42:13 STEP 1174 H= 293:30 +CPU= 47.023
  15:43:00 STEP 1175 H= 293:45 +CPU= 47.115
  15:43:59 STEP 1176 H= 294:00 +CPU= 57.858
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001176
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001176
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001176
  15:44:47 STEP 1177 H= 294:15 +CPU= 47.897
  15:45:35 STEP 1178 H= 294:30 +CPU= 47.726
  15:46:23 STEP 1179 H= 294:45 +CPU= 47.292
  15:47:12 STEP 1180 H= 295:00 +CPU= 48.898
  15:48:01 STEP 1181 H= 295:15 +CPU= 48.001
  15:48:49 STEP 1182 H= 295:30 +CPU= 47.840
  15:49:37 STEP 1183 H= 295:45 +CPU= 47.530
  15:50:26 STEP 1184 H= 296:00 +CPU= 48.460
  15:51:14 STEP 1185 H= 296:15 +CPU= 47.580
  15:52:02 STEP 1186 H= 296:30 +CPU= 48.415
  15:52:50 STEP 1187 H= 296:45 +CPU= 47.578
  15:53:50 STEP 1188 H= 297:00 +CPU= 59.007
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001188
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001188
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001188
  15:54:38 STEP 1189 H= 297:15 +CPU= 47.914
  15:55:25 STEP 1190 H= 297:30 +CPU= 46.115
  15:56:12 STEP 1191 H= 297:45 +CPU= 47.371
  15:57:01 STEP 1192 H= 298:00 +CPU= 47.813
  15:57:48 STEP 1193 H= 298:15 +CPU= 47.089
  15:58:37 STEP 1194 H= 298:30 +CPU= 48.271
  15:59:26 STEP 1195 H= 298:45 +CPU= 48.484
  16:00:16 STEP 1196 H= 299:00 +CPU= 49.543
  16:01:05 STEP 1197 H= 299:15 +CPU= 48.076
  16:01:51 STEP 1198 H= 299:30 +CPU= 46.381
  16:02:40 STEP 1199 H= 299:45 +CPU= 47.902
  16:03:46 STEP 1200 H= 300:00 +CPU= 65.528
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001200
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001200
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001200
  16:04:33 STEP 1201 H= 300:15 +CPU= 47.027
  16:05:20 STEP 1202 H= 300:30 +CPU= 46.311
  16:06:08 STEP 1203 H= 300:45 +CPU= 47.280
  16:06:58 STEP 1204 H= 301:00 +CPU= 49.807
  16:07:45 STEP 1205 H= 301:15 +CPU= 46.717
  16:08:32 STEP 1206 H= 301:30 +CPU= 46.906
  16:09:20 STEP 1207 H= 301:45 +CPU= 47.702
  16:10:10 STEP 1208 H= 302:00 +CPU= 48.933
  16:10:58 STEP 1209 H= 302:15 +CPU= 48.438
  16:11:46 STEP 1210 H= 302:30 +CPU= 46.654
  16:12:34 STEP 1211 H= 302:45 +CPU= 47.123
  16:13:32 STEP 1212 H= 303:00 +CPU= 57.634
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001212
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001212
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001212
  16:14:21 STEP 1213 H= 303:15 +CPU= 47.759
  16:15:10 STEP 1214 H= 303:30 +CPU= 48.579
  16:15:56 STEP 1215 H= 303:45 +CPU= 46.216
  16:16:45 STEP 1216 H= 304:00 +CPU= 48.602
  16:17:34 STEP 1217 H= 304:15 +CPU= 48.272
  16:18:23 STEP 1218 H= 304:30 +CPU= 48.480
  16:19:11 STEP 1219 H= 304:45 +CPU= 47.711
  16:20:01 STEP 1220 H= 305:00 +CPU= 49.304
  16:20:47 STEP 1221 H= 305:15 +CPU= 46.243
  16:21:35 STEP 1222 H= 305:30 +CPU= 47.143
  16:22:23 STEP 1223 H= 305:45 +CPU= 47.965
  16:23:21 STEP 1224 H= 306:00 +CPU= 57.103
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001224
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001224
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001224
  16:24:08 STEP 1225 H= 306:15 +CPU= 46.711
  16:24:56 STEP 1226 H= 306:30 +CPU= 47.355
  16:25:42 STEP 1227 H= 306:45 +CPU= 45.983
  16:26:30 STEP 1228 H= 307:00 +CPU= 47.351
  16:27:18 STEP 1229 H= 307:15 +CPU= 46.846
  16:28:06 STEP 1230 H= 307:30 +CPU= 46.992
  16:28:54 STEP 1231 H= 307:45 +CPU= 47.713
  16:29:42 STEP 1232 H= 308:00 +CPU= 48.391
  16:30:30 STEP 1233 H= 308:15 +CPU= 47.380
  16:31:18 STEP 1234 H= 308:30 +CPU= 47.688
  16:32:06 STEP 1235 H= 308:45 +CPU= 47.804
  16:33:03 STEP 1236 H= 309:00 +CPU= 56.552
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001236
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001236
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001236
  16:33:52 STEP 1237 H= 309:15 +CPU= 47.921
  16:34:39 STEP 1238 H= 309:30 +CPU= 47.222
  16:35:29 STEP 1239 H= 309:45 +CPU= 48.919
  16:36:17 STEP 1240 H= 310:00 +CPU= 48.061
  16:37:06 STEP 1241 H= 310:15 +CPU= 48.771
  16:37:55 STEP 1242 H= 310:30 +CPU= 48.346
  16:38:43 STEP 1243 H= 310:45 +CPU= 47.816
  16:39:32 STEP 1244 H= 311:00 +CPU= 48.931
  16:40:20 STEP 1245 H= 311:15 +CPU= 47.327
  16:41:10 STEP 1246 H= 311:30 +CPU= 49.247
  16:41:57 STEP 1247 H= 311:45 +CPU= 47.432
  16:43:00 STEP 1248 H= 312:00 +CPU= 61.932
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001248
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001248
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_12.zip
Uploading the intermediate file: upload_file_12.zip
  16:43:48 STEP 1249 H= 312:15 +CPU= 46.636
  16:44:36 STEP 1250 H= 312:30 +CPU= 47.413
  16:45:24 STEP 1251 H= 312:45 +CPU= 47.930
  16:46:13 STEP 1252 H= 313:00 +CPU= 47.983
  16:47:02 STEP 1253 H= 313:15 +CPU= 48.588
  16:47:51 STEP 1254 H= 313:30 +CPU= 48.637
  16:48:39 STEP 1255 H= 313:45 +CPU= 47.204
  16:49:29 STEP 1256 H= 314:00 +CPU= 49.833
  16:50:18 STEP 1257 H= 314:15 +CPU= 48.410
  16:51:06 STEP 1258 H= 314:30 +CPU= 47.795
  16:51:56 STEP 1259 H= 314:45 +CPU= 48.752
  16:52:52 STEP 1260 H= 315:00 +CPU= 56.088
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001260
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001260
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001260
  16:53:40 STEP 1261 H= 315:15 +CPU= 47.071
  16:54:28 STEP 1262 H= 315:30 +CPU= 48.060
  16:55:15 STEP 1263 H= 315:45 +CPU= 46.381
  16:56:04 STEP 1264 H= 316:00 +CPU= 48.340
  16:56:53 STEP 1265 H= 316:15 +CPU= 48.265
  16:57:40 STEP 1266 H= 316:30 +CPU= 46.803
  16:58:27 STEP 1267 H= 316:45 +CPU= 46.696
  16:59:15 STEP 1268 H= 317:00 +CPU= 47.813
  17:00:03 STEP 1269 H= 317:15 +CPU= 47.094
  17:00:50 STEP 1270 H= 317:30 +CPU= 47.242
  17:01:38 STEP 1271 H= 317:45 +CPU= 47.350
  17:02:35 STEP 1272 H= 318:00 +CPU= 56.905
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001272
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001272
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001272
  17:03:24 STEP 1273 H= 318:15 +CPU= 48.259
  17:04:12 STEP 1274 H= 318:30 +CPU= 47.657
  17:05:00 STEP 1275 H= 318:45 +CPU= 47.692
  17:05:50 STEP 1276 H= 319:00 +CPU= 49.204
  17:06:38 STEP 1277 H= 319:15 +CPU= 47.397
  17:07:26 STEP 1278 H= 319:30 +CPU= 47.550
  17:08:14 STEP 1279 H= 319:45 +CPU= 47.618
  17:09:03 STEP 1280 H= 320:00 +CPU= 48.779
  17:09:52 STEP 1281 H= 320:15 +CPU= 48.328
  17:10:40 STEP 1282 H= 320:30 +CPU= 47.846
  17:11:29 STEP 1283 H= 320:45 +CPU= 49.144
  17:12:28 STEP 1284 H= 321:00 +CPU= 57.971
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001284
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001284
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001284
  17:13:15 STEP 1285 H= 321:15 +CPU= 46.611
  17:14:02 STEP 1286 H= 321:30 +CPU= 47.025
  17:14:50 STEP 1287 H= 321:45 +CPU= 46.812
  17:15:38 STEP 1288 H= 322:00 +CPU= 47.729
  17:16:26 STEP 1289 H= 322:15 +CPU= 47.847
  17:17:13 STEP 1290 H= 322:30 +CPU= 47.339
  17:18:02 STEP 1291 H= 322:45 +CPU= 47.800
  17:18:49 STEP 1292 H= 323:00 +CPU= 47.333
  17:19:37 STEP 1293 H= 323:15 +CPU= 46.851
  17:20:24 STEP 1294 H= 323:30 +CPU= 47.182
  17:21:11 STEP 1295 H= 323:45 +CPU= 46.655
  17:22:14 STEP 1296 H= 324:00 +CPU= 62.020
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001296
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001296
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001296
  17:23:03 STEP 1297 H= 324:15 +CPU= 48.422
  17:23:51 STEP 1298 H= 324:30 +CPU= 47.825
  17:24:41 STEP 1299 H= 324:45 +CPU= 49.200
  17:25:29 STEP 1300 H= 325:00 +CPU= 47.851
  17:26:17 STEP 1301 H= 325:15 +CPU= 47.137
  17:27:05 STEP 1302 H= 325:30 +CPU= 47.104
  17:27:54 STEP 1303 H= 325:45 +CPU= 48.192
  17:28:43 STEP 1304 H= 326:00 +CPU= 48.031
  17:29:30 STEP 1305 H= 326:15 +CPU= 46.790
  17:30:18 STEP 1306 H= 326:30 +CPU= 47.854
  17:31:07 STEP 1307 H= 326:45 +CPU= 47.892
  17:32:04 STEP 1308 H= 327:00 +CPU= 56.653
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001308
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001308
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001308
  17:32:51 STEP 1309 H= 327:15 +CPU= 46.845
  17:33:38 STEP 1310 H= 327:30 +CPU= 46.742
  17:34:26 STEP 1311 H= 327:45 +CPU= 47.491
  17:35:14 STEP 1312 H= 328:00 +CPU= 47.544
  17:36:00 STEP 1313 H= 328:15 +CPU= 46.053
  17:36:48 STEP 1314 H= 328:30 +CPU= 47.059
  17:37:34 STEP 1315 H= 328:45 +CPU= 45.876
  17:38:22 STEP 1316 H= 329:00 +CPU= 47.406
  17:39:10 STEP 1317 H= 329:15 +CPU= 47.227
  17:39:57 STEP 1318 H= 329:30 +CPU= 46.759
  17:40:44 STEP 1319 H= 329:45 +CPU= 46.657
  17:41:42 STEP 1320 H= 330:00 +CPU= 58.025
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001320
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001320
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001320
  17:42:31 STEP 1321 H= 330:15 +CPU= 47.394
  17:43:20 STEP 1322 H= 330:30 +CPU= 48.769
  17:44:09 STEP 1323 H= 330:45 +CPU= 48.148
  17:44:57 STEP 1324 H= 331:00 +CPU= 48.439
  17:45:45 STEP 1325 H= 331:15 +CPU= 47.710
  17:46:35 STEP 1326 H= 331:30 +CPU= 48.855
  17:47:23 STEP 1327 H= 331:45 +CPU= 47.541
  17:48:11 STEP 1328 H= 332:00 +CPU= 48.120
  17:48:59 STEP 1329 H= 332:15 +CPU= 46.955
  17:49:47 STEP 1330 H= 332:30 +CPU= 47.772
  17:50:35 STEP 1331 H= 332:45 +CPU= 47.738
  17:51:32 STEP 1332 H= 333:00 +CPU= 56.154
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001332
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001332
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001332
  17:52:19 STEP 1333 H= 333:15 +CPU= 46.681
  17:53:06 STEP 1334 H= 333:30 +CPU= 46.928
  17:53:54 STEP 1335 H= 333:45 +CPU= 47.489
  17:54:41 STEP 1336 H= 334:00 +CPU= 46.843
  17:55:28 STEP 1337 H= 334:15 +CPU= 46.426
  17:56:16 STEP 1338 H= 334:30 +CPU= 47.219
  17:57:02 STEP 1339 H= 334:45 +CPU= 46.378
  17:57:50 STEP 1340 H= 335:00 +CPU= 46.863
  17:58:37 STEP 1341 H= 335:15 +CPU= 47.329
  17:59:25 STEP 1342 H= 335:30 +CPU= 47.307
  18:00:12 STEP 1343 H= 335:45 +CPU= 46.812
  18:01:16 STEP 1344 H= 336:00 +CPU= 63.054
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001344
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001344
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_13.zip
Uploading the intermediate file: upload_file_13.zip
  18:02:04 STEP 1345 H= 336:15 +CPU= 47.474
  18:02:54 STEP 1346 H= 336:30 +CPU= 48.921
  18:03:41 STEP 1347 H= 336:45 +CPU= 46.093
  18:04:30 STEP 1348 H= 337:00 +CPU= 48.927
  18:05:19 STEP 1349 H= 337:15 +CPU= 47.899
  18:06:07 STEP 1350 H= 337:30 +CPU= 48.289
  18:06:56 STEP 1351 H= 337:45 +CPU= 47.620
  18:07:44 STEP 1352 H= 338:00 +CPU= 47.616
  18:08:31 STEP 1353 H= 338:15 +CPU= 47.205
  18:09:20 STEP 1354 H= 338:30 +CPU= 48.146
  18:10:09 STEP 1355 H= 338:45 +CPU= 48.604
  18:11:04 STEP 1356 H= 339:00 +CPU= 54.515
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001356
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001356
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001356
  18:11:51 STEP 1357 H= 339:15 +CPU= 46.671
  18:12:39 STEP 1358 H= 339:30 +CPU= 47.686
  18:13:27 STEP 1359 H= 339:45 +CPU= 47.673
  18:14:15 STEP 1360 H= 340:00 +CPU= 47.859
  18:15:02 STEP 1361 H= 340:15 +CPU= 46.336
  18:15:49 STEP 1362 H= 340:30 +CPU= 47.088
  18:16:37 STEP 1363 H= 340:45 +CPU= 47.053
  18:17:25 STEP 1364 H= 341:00 +CPU= 47.809
  18:18:13 STEP 1365 H= 341:15 +CPU= 47.900
  18:19:02 STEP 1366 H= 341:30 +CPU= 48.421
  18:19:50 STEP 1367 H= 341:45 +CPU= 47.315
  18:20:49 STEP 1368 H= 342:00 +CPU= 58.096
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001368
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001368
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001368
  18:21:37 STEP 1369 H= 342:15 +CPU= 47.838
  18:22:25 STEP 1370 H= 342:30 +CPU= 48.131
  18:23:15 STEP 1371 H= 342:45 +CPU= 48.884
  18:24:03 STEP 1372 H= 343:00 +CPU= 48.335
  18:24:51 STEP 1373 H= 343:15 +CPU= 47.295
  18:25:39 STEP 1374 H= 343:30 +CPU= 47.324
  18:26:27 STEP 1375 H= 343:45 +CPU= 47.671
  18:27:14 STEP 1376 H= 344:00 +CPU= 47.048
  18:28:02 STEP 1377 H= 344:15 +CPU= 47.061
  18:28:50 STEP 1378 H= 344:30 +CPU= 47.436
  18:29:37 STEP 1379 H= 344:45 +CPU= 46.731
  18:30:36 STEP 1380 H= 345:00 +CPU= 58.033
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001380
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001380
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001380
  18:31:23 STEP 1381 H= 345:15 +CPU= 47.111
  18:32:11 STEP 1382 H= 345:30 +CPU= 47.552
  18:32:58 STEP 1383 H= 345:45 +CPU= 46.680
  18:33:46 STEP 1384 H= 346:00 +CPU= 47.708
  18:34:34 STEP 1385 H= 346:15 +CPU= 47.687
  18:35:23 STEP 1386 H= 346:30 +CPU= 47.696
  18:36:11 STEP 1387 H= 346:45 +CPU= 48.134
  18:37:00 STEP 1388 H= 347:00 +CPU= 48.287
  18:37:48 STEP 1389 H= 347:15 +CPU= 47.668
  18:38:35 STEP 1390 H= 347:30 +CPU= 46.993
  18:39:23 STEP 1391 H= 347:45 +CPU= 47.203
  18:40:28 STEP 1392 H= 348:00 +CPU= 64.066
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001392
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001392
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001392
  18:41:16 STEP 1393 H= 348:15 +CPU= 47.819
  18:42:03 STEP 1394 H= 348:30 +CPU= 45.729
  18:42:50 STEP 1395 H= 348:45 +CPU= 46.676
  18:43:38 STEP 1396 H= 349:00 +CPU= 47.125
  18:44:25 STEP 1397 H= 349:15 +CPU= 46.718
  18:45:13 STEP 1398 H= 349:30 +CPU= 47.634
  18:45:59 STEP 1399 H= 349:45 +CPU= 46.005
  18:46:49 STEP 1400 H= 350:00 +CPU= 48.685
  18:47:37 STEP 1401 H= 350:15 +CPU= 48.074
  18:48:27 STEP 1402 H= 350:30 +CPU= 49.238
  18:49:19 STEP 1403 H= 350:45 +CPU= 52.225
  18:50:24 STEP 1404 H= 351:00 +CPU= 63.718
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001404
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001404
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001404
  18:51:17 STEP 1405 H= 351:15 +CPU= 52.559
  18:52:07 STEP 1406 H= 351:30 +CPU= 49.967
  18:53:02 STEP 1407 H= 351:45 +CPU= 54.588
  18:53:55 STEP 1408 H= 352:00 +CPU= 52.458
  18:54:47 STEP 1409 H= 352:15 +CPU= 51.731
  18:55:39 STEP 1410 H= 352:30 +CPU= 51.541
  18:56:30 STEP 1411 H= 352:45 +CPU= 50.460
  18:57:24 STEP 1412 H= 353:00 +CPU= 53.748
  18:58:11 STEP 1413 H= 353:15 +CPU= 46.141
  18:58:58 STEP 1414 H= 353:30 +CPU= 46.721
  18:59:45 STEP 1415 H= 353:45 +CPU= 46.617
  19:00:40 STEP 1416 H= 354:00 +CPU= 55.323
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001416
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001416
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001416
  19:01:25 STEP 1417 H= 354:15 +CPU= 44.133
  19:02:12 STEP 1418 H= 354:30 +CPU= 46.633
  19:03:00 STEP 1419 H= 354:45 +CPU= 48.099
  19:03:51 STEP 1420 H= 355:00 +CPU= 50.207
  19:04:41 STEP 1421 H= 355:15 +CPU= 49.320
  19:05:30 STEP 1422 H= 355:30 +CPU= 49.025
  19:06:21 STEP 1423 H= 355:45 +CPU= 50.675
  19:07:13 STEP 1424 H= 356:00 +CPU= 51.745
  19:08:03 STEP 1425 H= 356:15 +CPU= 49.402
  19:08:52 STEP 1426 H= 356:30 +CPU= 48.351
  19:09:42 STEP 1427 H= 356:45 +CPU= 49.544
  19:10:43 STEP 1428 H= 357:00 +CPU= 60.328
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001428
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001428
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001428
  19:11:32 STEP 1429 H= 357:15 +CPU= 49.328
  19:12:22 STEP 1430 H= 357:30 +CPU= 49.170
  19:13:12 STEP 1431 H= 357:45 +CPU= 49.733
  19:14:03 STEP 1432 H= 358:00 +CPU= 50.697
  19:14:52 STEP 1433 H= 358:15 +CPU= 48.809
  19:15:39 STEP 1434 H= 358:30 +CPU= 46.217
  19:16:25 STEP 1435 H= 358:45 +CPU= 45.550
  19:17:15 STEP 1436 H= 359:00 +CPU= 50.021
  19:18:04 STEP 1437 H= 359:15 +CPU= 47.844
  19:18:53 STEP 1438 H= 359:30 +CPU= 49.419
  19:19:45 STEP 1439 H= 359:45 +CPU= 51.253
  19:20:46 STEP 1440 H= 360:00 +CPU= 60.685
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.1414570909E-22      I       0.1340555168E-04      I
ICloud fractionI      0.0000000000E+00      I       0.1100380847E+00      I
I  Relat. Hum. I      0.3579853425E-01      I       0.3577200146E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1709820995E+04      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    303511204 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS505852.0 Mbytes/s, TOTAL TIME=    0.0179( 3.4%)
 19:20:46 STEP 1440 H= 360:00 +CPU= 60.685
 END CNT3
 NSTEP =  1440 CNT0   000000000  45.156  45.156  45.568   0.991               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        43246/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    43246 ************   109.9     0.0   109.8     0.0   266.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.99      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                 1562    9547.4    9547.4     18.90      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    1900.1    1900.1      3.47      0.00
  10 SCAN2M - PHYSICS                             1441   24941.7   24941.7     45.54      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1486.8    1486.8      0.23      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63     120.8     120.8      0.01      0.00
  14 SUINIF                                          1    7942.9    7942.9      0.01      0.00
  17 GRIDFPOS IN CNT4                              121      56.3      56.3      0.01      0.00
  18 SUSPECG                                         1    4084.8    4084.8      0.01      0.00
  19 SUSPEC                                          1    4095.7    4095.7      0.01      0.00
  24 SUGRIDU                                         1    3208.5    3208.5      0.00      0.00
  25 SPECRT                                          1     487.7     487.7      0.00      0.00
  26 SUGRIDF                                         1     150.8     150.8      0.00      0.00
  27 RESTART FILES - WRITING                        30    5996.7    5996.7      0.23      0.00
  28 RESTART FILES - READING                         1       0.1       0.1      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                               121   10525.1   10525.1      1.61      0.00
  31 POSDDH IN STEPO                                 7       1.3       1.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441   10865.9   10865.9     19.84      0.00
  39 SU0YOMB                                         1     510.7     510.7      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      40.3      40.3      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440    1055.1    1055.1      1.93      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440     253.3     253.3      0.46      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.7       0.7      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     744.7     744.7      2.95      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     436.3     436.3      1.66      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     150.5     150.5      0.57      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     235.1     235.1      0.93      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      61.7      61.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125     121.0     121.0      0.48      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      76.4      76.4      0.29      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125     186.2     186.2      0.74      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      81.6      81.6      0.31      0.00
 400 GSTATS                                     167734       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                156066       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    78916.3 CPU TIME   77537.8 VECTOR TIME    77537.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       16.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/steve/slots/7/ICMGGh7zg+001440
Moving to projects directory: /home/steve/slots/7/ICMSHh7zg+001440
Moving to projects directory: /home/steve/slots/7/ICMUAh7zg+001440
Adding to the zip: /home/steve/slots/7/NODE.001_01
Adding to the zip: /home/steve/slots/7/ifs.stat
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMUAh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMSHh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12283511/ICMGGh7zg+001404
Zipping up the final file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
19:24:19 (974): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0_r814855303_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 00:55:17 1536520 22435090 oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0 1,295,100 77,538 0.0599
08 Jun 2024 18:42:10 1536520 22435090 oifs_43r3_bl_a1as_2016092300_20_1017_12283511_0 864,000 54,177 0.0627


©2024 climateprediction.net