climateprediction.net home page
Task 22309156

Task 22309156

Name oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0
Workunit 12208976
Created 13 Feb 2023, 16:02:29 UTC
Sent 13 Feb 2023, 16:10:29 UTC
Report deadline 14 Apr 2023, 16:10:29 UTC
Received 14 Feb 2023, 6:47:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1536520
Run time 13 hours 48 min 59 sec
CPU time 13 hours 30 min 4 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 7.36 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 5,058.16 MB
Peak swap size 5,930.79 MB
Peak disk usage 1,370.25 MB
Stderr
<core_client_version>7.22.0</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
0213:101052:1676304652:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at 0x1)
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:steve-System-Product-Name:1:1:167:167] [20230213:101052:1676304652:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:12:08 STEP    0 H=   0:00 +CPU= 71.340
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000000
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000000
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000000
  10:13:28 STEP    1 H=   0:15 +CPU= 77.132
  10:14:41 STEP    2 H=   0:30 +CPU= 71.959
  10:15:51 STEP    3 H=   0:45 +CPU= 69.270
  10:17:08 STEP    4 H=   1:00 +CPU= 75.700
  10:18:24 STEP    5 H=   1:15 +CPU= 75.152
  10:19:38 STEP    6 H=   1:30 +CPU= 72.133
  10:20:56 STEP    7 H=   1:45 +CPU= 77.607
  10:22:14 STEP    8 H=   2:00 +CPU= 76.201
  10:23:29 STEP    9 H=   2:15 +CPU= 73.527
  10:24:50 STEP   10 H=   2:30 +CPU= 79.892
  10:26:08 STEP   11 H=   2:45 +CPU= 76.979
  10:27:49 STEP   12 H=   3:00 +CPU= 99.294
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000012
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000012
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000012
  10:29:13 STEP   13 H=   3:15 +CPU= 81.000
  10:30:35 STEP   14 H=   3:30 +CPU= 80.662
  10:32:02 STEP   15 H=   3:45 +CPU= 85.318
  10:33:28 STEP   16 H=   4:00 +CPU= 84.859
  10:34:56 STEP   17 H=   4:15 +CPU= 87.054
  10:36:23 STEP   18 H=   4:30 +CPU= 85.522
  10:37:55 STEP   19 H=   4:45 +CPU= 91.346
  10:39:19 STEP   20 H=   5:00 +CPU= 83.372
  10:40:46 STEP   21 H=   5:15 +CPU= 85.665
  10:42:18 STEP   22 H=   5:30 +CPU= 90.680
  10:43:39 STEP   23 H=   5:45 +CPU= 80.097
  10:45:27 STEP   24 H=   6:00 +CPU=105.925
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000024
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000024
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000024
  10:47:00 STEP   25 H=   6:15 +CPU= 91.984
  10:48:21 STEP   26 H=   6:30 +CPU= 79.716
  10:49:44 STEP   27 H=   6:45 +CPU= 81.715
  10:51:08 STEP   28 H=   7:00 +CPU= 82.656
  10:52:33 STEP   29 H=   7:15 +CPU= 83.923
  10:53:59 STEP   30 H=   7:30 +CPU= 84.748
  10:55:24 STEP   31 H=   7:45 +CPU= 84.152
  10:56:52 STEP   32 H=   8:00 +CPU= 86.569
  10:58:23 STEP   33 H=   8:15 +CPU= 89.963
  10:59:50 STEP   34 H=   8:30 +CPU= 84.817
  11:01:21 STEP   35 H=   8:45 +CPU= 90.285
  11:03:15 STEP   36 H=   9:00 +CPU=112.017
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000036
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000036
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000036
  11:04:34 STEP   37 H=   9:15 +CPU= 78.427
  11:06:00 STEP   38 H=   9:30 +CPU= 84.150
  11:07:30 STEP   39 H=   9:45 +CPU= 89.554
  11:08:59 STEP   40 H=  10:00 +CPU= 87.278
  11:10:26 STEP   41 H=  10:15 +CPU= 85.937
  11:11:54 STEP   42 H=  10:30 +CPU= 86.318
  11:13:18 STEP   43 H=  10:45 +CPU= 83.165
  11:14:46 STEP   44 H=  11:00 +CPU= 86.627
  11:16:14 STEP   45 H=  11:15 +CPU= 87.656
  11:17:40 STEP   46 H=  11:30 +CPU= 84.652
  11:19:08 STEP   47 H=  11:45 +CPU= 86.203
  11:21:06 STEP   48 H=  12:00 +CPU=115.104
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000048
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000048
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000048
  11:22:37 STEP   49 H=  12:15 +CPU= 89.218
  11:24:02 STEP   50 H=  12:30 +CPU= 83.534
  11:25:32 STEP   51 H=  12:45 +CPU= 88.779
  11:27:04 STEP   52 H=  13:00 +CPU= 91.108
  11:28:24 STEP   53 H=  13:15 +CPU= 77.835
  11:29:53 STEP   54 H=  13:30 +CPU= 87.688
  11:31:16 STEP   55 H=  13:45 +CPU= 82.110
  11:32:43 STEP   56 H=  14:00 +CPU= 84.923
  11:34:10 STEP   57 H=  14:15 +CPU= 86.092
  11:35:36 STEP   58 H=  14:30 +CPU= 84.979
  11:37:02 STEP   59 H=  14:45 +CPU= 84.842
  11:38:42 STEP   60 H=  15:00 +CPU= 98.194
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000060
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000060
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000060
  11:40:18 STEP   61 H=  15:15 +CPU= 93.986
  11:41:42 STEP   62 H=  15:30 +CPU= 82.596
  11:43:10 STEP   63 H=  15:45 +CPU= 86.614
  11:44:33 STEP   64 H=  16:00 +CPU= 81.320
  11:45:56 STEP   65 H=  16:15 +CPU= 81.530
  11:47:25 STEP   66 H=  16:30 +CPU= 87.172
  11:48:51 STEP   67 H=  16:45 +CPU= 85.086
  11:50:16 STEP   68 H=  17:00 +CPU= 83.792
  11:51:45 STEP   69 H=  17:15 +CPU= 87.490
  11:53:08 STEP   70 H=  17:30 +CPU= 81.788
  11:54:35 STEP   71 H=  17:45 +CPU= 84.939
  11:56:30 STEP   72 H=  18:00 +CPU=113.220
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000072
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000072
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000072
  11:57:54 STEP   73 H=  18:15 +CPU= 82.447
  11:59:26 STEP   74 H=  18:30 +CPU= 91.029
  12:00:54 STEP   75 H=  18:45 +CPU= 87.020
  12:02:18 STEP   76 H=  19:00 +CPU= 82.730
  12:03:51 STEP   77 H=  19:15 +CPU= 90.749
  12:05:15 STEP   78 H=  19:30 +CPU= 83.167
  12:06:44 STEP   79 H=  19:45 +CPU= 87.338
  12:08:13 STEP   80 H=  20:00 +CPU= 87.450
  12:09:41 STEP   81 H=  20:15 +CPU= 86.286
  12:11:08 STEP   82 H=  20:30 +CPU= 86.592
  12:12:34 STEP   83 H=  20:45 +CPU= 84.343
  12:14:12 STEP   84 H=  21:00 +CPU= 96.418
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000084
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000084
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000084
  12:15:36 STEP   85 H=  21:15 +CPU= 82.161
  12:17:08 STEP   86 H=  21:30 +CPU= 90.908
  12:18:31 STEP   87 H=  21:45 +CPU= 81.271
  12:20:02 STEP   88 H=  22:00 +CPU= 89.538
  12:21:27 STEP   89 H=  22:15 +CPU= 83.234
  12:22:55 STEP   90 H=  22:30 +CPU= 86.729
  12:24:26 STEP   91 H=  22:45 +CPU= 88.937
  12:25:53 STEP   92 H=  23:00 +CPU= 85.748
  12:27:20 STEP   93 H=  23:15 +CPU= 84.617
  12:28:49 STEP   94 H=  23:30 +CPU= 86.252
  12:30:11 STEP   95 H=  23:45 +CPU= 79.029
  12:32:18 STEP   96 H=  24:00 +CPU=123.141
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000096
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000096
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000096
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000000
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000000
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000000
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000012
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000012
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000012
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000024
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000024
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000024
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000036
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000036
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000036
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000048
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000048
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000048
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000060
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000060
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000060
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000072
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000072
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000072
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000084
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000084
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000084
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0_r669128754_0.zip
Uploading the intermediate file: upload_file_0.zip
  12:33:43 STEP   97 H=  24:15 +CPU= 80.546
  12:35:12 STEP   98 H=  24:30 +CPU= 88.007
  12:36:40 STEP   99 H=  24:45 +CPU= 86.265
  12:38:03 STEP  100 H=  25:00 +CPU= 81.447
  12:39:31 STEP  101 H=  25:15 +CPU= 86.985
  12:41:04 STEP  102 H=  25:30 +CPU= 90.922
  12:42:25 STEP  103 H=  25:45 +CPU= 79.835
  12:43:58 STEP  104 H=  26:00 +CPU= 88.322
  12:45:29 STEP  105 H=  26:15 +CPU= 89.312
  12:46:52 STEP  106 H=  26:30 +CPU= 81.685
  12:48:20 STEP  107 H=  26:45 +CPU= 86.078
  12:50:08 STEP  108 H=  27:00 +CPU=107.493
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000108
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000108
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000108
  12:51:36 STEP  109 H=  27:15 +CPU= 85.696
  12:53:04 STEP  110 H=  27:30 +CPU= 86.642
  12:54:28 STEP  111 H=  27:45 +CPU= 83.507
  12:55:57 STEP  112 H=  28:00 +CPU= 87.922
  12:57:26 STEP  113 H=  28:15 +CPU= 87.321
  12:58:49 STEP  114 H=  28:30 +CPU= 81.571
  13:00:17 STEP  115 H=  28:45 +CPU= 87.353
  13:01:42 STEP  116 H=  29:00 +CPU= 83.373
  13:03:06 STEP  117 H=  29:15 +CPU= 83.094
  13:04:33 STEP  118 H=  29:30 +CPU= 85.815
  13:05:59 STEP  119 H=  29:45 +CPU= 84.447
  13:07:51 STEP  120 H=  30:00 +CPU=109.812
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000120
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000120
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000120
  13:09:21 STEP  121 H=  30:15 +CPU= 89.383
  13:10:45 STEP  122 H=  30:30 +CPU= 82.184
  13:12:15 STEP  123 H=  30:45 +CPU= 88.425
  13:13:44 STEP  124 H=  31:00 +CPU= 88.125
  13:15:17 STEP  125 H=  31:15 +CPU= 91.332
  13:16:51 STEP  126 H=  31:30 +CPU= 93.285
  13:18:16 STEP  127 H=  31:45 +CPU= 83.018
  13:19:46 STEP  128 H=  32:00 +CPU= 88.499
  13:21:18 STEP  129 H=  32:15 +CPU= 91.199
  13:22:42 STEP  130 H=  32:30 +CPU= 82.734
  13:24:16 STEP  131 H=  32:45 +CPU= 92.757
  13:26:00 STEP  132 H=  33:00 +CPU=101.951
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000132
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000132
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000132
  13:27:27 STEP  133 H=  33:15 +CPU= 85.855
  13:28:56 STEP  134 H=  33:30 +CPU= 87.367
  13:30:24 STEP  135 H=  33:45 +CPU= 86.680
  13:31:54 STEP  136 H=  34:00 +CPU= 88.859
  13:33:24 STEP  137 H=  34:15 +CPU= 88.088
  13:34:51 STEP  138 H=  34:30 +CPU= 85.723
  13:36:17 STEP  139 H=  34:45 +CPU= 85.499
  13:37:47 STEP  140 H=  35:00 +CPU= 87.540
  13:39:13 STEP  141 H=  35:15 +CPU= 84.018
  13:40:46 STEP  142 H=  35:30 +CPU= 91.419
  13:42:16 STEP  143 H=  35:45 +CPU= 88.949
  13:44:16 STEP  144 H=  36:00 +CPU=116.912
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000144
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000144
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000144
  13:45:41 STEP  145 H=  36:15 +CPU= 84.023
  13:47:06 STEP  146 H=  36:30 +CPU= 83.252
  13:48:33 STEP  147 H=  36:45 +CPU= 85.259
  13:49:57 STEP  148 H=  37:00 +CPU= 83.041
  13:51:19 STEP  149 H=  37:15 +CPU= 80.503
  13:52:51 STEP  150 H=  37:30 +CPU= 90.153
  13:54:17 STEP  151 H=  37:45 +CPU= 84.647
  13:55:43 STEP  152 H=  38:00 +CPU= 85.546
  13:57:11 STEP  153 H=  38:15 +CPU= 86.650
  13:58:42 STEP  154 H=  38:30 +CPU= 89.729
  14:00:05 STEP  155 H=  38:45 +CPU= 80.975
  14:01:50 STEP  156 H=  39:00 +CPU=103.467
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000156
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000156
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000156
  14:03:19 STEP  157 H=  39:15 +CPU= 87.806
  14:04:50 STEP  158 H=  39:30 +CPU= 89.796
  14:06:18 STEP  159 H=  39:45 +CPU= 86.077
  14:07:43 STEP  160 H=  40:00 +CPU= 83.740
  14:09:10 STEP  161 H=  40:15 +CPU= 85.266
  14:10:37 STEP  162 H=  40:30 +CPU= 85.794
  14:12:06 STEP  163 H=  40:45 +CPU= 87.949
  14:13:32 STEP  164 H=  41:00 +CPU= 84.713
  14:15:00 STEP  165 H=  41:15 +CPU= 87.124
  14:16:30 STEP  166 H=  41:30 +CPU= 88.461
  14:17:50 STEP  167 H=  41:45 +CPU= 78.815
  14:19:40 STEP  168 H=  42:00 +CPU=108.310
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000168
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000168
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000168
  14:21:10 STEP  169 H=  42:15 +CPU= 88.917
  14:22:37 STEP  170 H=  42:30 +CPU= 86.039
  14:23:59 STEP  171 H=  42:45 +CPU= 79.911
  14:25:34 STEP  172 H=  43:00 +CPU= 94.054
  14:26:59 STEP  173 H=  43:15 +CPU= 84.276
  14:28:24 STEP  174 H=  43:30 +CPU= 84.008
  14:29:52 STEP  175 H=  43:45 +CPU= 86.453
  14:31:24 STEP  176 H=  44:00 +CPU= 90.928
  14:32:45 STEP  177 H=  44:15 +CPU= 79.746
  14:34:10 STEP  178 H=  44:30 +CPU= 84.324
  14:35:41 STEP  179 H=  44:45 +CPU= 88.944
  14:37:21 STEP  180 H=  45:00 +CPU= 98.871
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000180
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000180
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000180
  14:38:51 STEP  181 H=  45:15 +CPU= 88.508
  14:40:11 STEP  182 H=  45:30 +CPU= 78.902
  14:41:37 STEP  183 H=  45:45 +CPU= 84.473
  14:43:07 STEP  184 H=  46:00 +CPU= 88.961
  14:44:31 STEP  185 H=  46:15 +CPU= 82.899
  14:45:55 STEP  186 H=  46:30 +CPU= 81.856
  14:47:23 STEP  187 H=  46:45 +CPU= 83.591
  14:48:50 STEP  188 H=  47:00 +CPU= 84.272
  14:50:13 STEP  189 H=  47:15 +CPU= 81.742
  14:51:40 STEP  190 H=  47:30 +CPU= 85.409
  14:53:06 STEP  191 H=  47:45 +CPU= 85.191
  14:55:05 STEP  192 H=  48:00 +CPU=114.125
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000192
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000192
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000192
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000096
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000096
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000096
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000108
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000108
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000108
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000120
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000120
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000120
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000132
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000132
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000132
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000144
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000144
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000144
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000156
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000156
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000156
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000168
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000168
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000168
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000180
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000180
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000180
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0_r669128754_1.zip
Uploading the intermediate file: upload_file_1.zip
  14:56:32 STEP  193 H=  48:15 +CPU= 84.774
  14:58:01 STEP  194 H=  48:30 +CPU= 87.383
  14:59:37 STEP  195 H=  48:45 +CPU= 93.112
  15:00:58 STEP  196 H=  49:00 +CPU= 79.666
  15:02:28 STEP  197 H=  49:15 +CPU= 88.660
  15:03:59 STEP  198 H=  49:30 +CPU= 89.449
  15:05:26 STEP  199 H=  49:45 +CPU= 85.532
  15:07:03 STEP  200 H=  50:00 +CPU= 95.950
  15:08:33 STEP  201 H=  50:15 +CPU= 88.727
  15:10:02 STEP  202 H=  50:30 +CPU= 87.058
  15:11:31 STEP  203 H=  50:45 +CPU= 87.208
  15:13:13 STEP  204 H=  51:00 +CPU=100.650
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000204
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000204
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000204
  15:14:43 STEP  205 H=  51:15 +CPU= 88.219
  15:16:18 STEP  206 H=  51:30 +CPU= 93.623
  15:17:42 STEP  207 H=  51:45 +CPU= 83.372
  15:19:16 STEP  208 H=  52:00 +CPU= 93.169
  15:20:40 STEP  209 H=  52:15 +CPU= 82.114
  15:22:08 STEP  210 H=  52:30 +CPU= 86.768
  15:23:43 STEP  211 H=  52:45 +CPU= 93.339
  15:25:13 STEP  212 H=  53:00 +CPU= 88.250
  15:26:44 STEP  213 H=  53:15 +CPU= 89.742
  15:28:10 STEP  214 H=  53:30 +CPU= 84.809
  15:29:30 STEP  215 H=  53:45 +CPU= 78.968
  15:31:17 STEP  216 H=  54:00 +CPU=105.050
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000216
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000216
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000216
  15:32:42 STEP  217 H=  54:15 +CPU= 84.597
  15:34:09 STEP  218 H=  54:30 +CPU= 84.568
  15:35:38 STEP  219 H=  54:45 +CPU= 75.688
  15:36:59 STEP  220 H=  55:00 +CPU= 72.720
  15:38:23 STEP  221 H=  55:15 +CPU= 76.384
  15:39:51 STEP  222 H=  55:30 +CPU= 86.784
  15:41:09 STEP  223 H=  55:45 +CPU= 76.522
  15:42:35 STEP  224 H=  56:00 +CPU= 84.890
  15:44:01 STEP  225 H=  56:15 +CPU= 84.953
  15:45:19 STEP  226 H=  56:30 +CPU= 76.371
  15:46:39 STEP  227 H=  56:45 +CPU= 78.645
  15:48:25 STEP  228 H=  57:00 +CPU=103.921
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000228
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000228
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000228
  15:49:46 STEP  229 H=  57:15 +CPU= 79.475
  15:51:14 STEP  230 H=  57:30 +CPU= 86.662
  15:52:33 STEP  231 H=  57:45 +CPU= 77.424
  15:53:50 STEP  232 H=  58:00 +CPU= 75.882
  15:55:12 STEP  233 H=  58:15 +CPU= 80.817
  15:56:32 STEP  234 H=  58:30 +CPU= 78.284
  15:57:54 STEP  235 H=  58:45 +CPU= 80.803
  15:59:20 STEP  236 H=  59:00 +CPU= 84.604
  16:00:41 STEP  237 H=  59:15 +CPU= 79.681
  16:02:04 STEP  238 H=  59:30 +CPU= 80.787
  16:03:26 STEP  239 H=  59:45 +CPU= 80.912
  16:05:12 STEP  240 H=  60:00 +CPU=103.809
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000240
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000240
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000240
  16:06:37 STEP  241 H=  60:15 +CPU= 83.520
  16:07:59 STEP  242 H=  60:30 +CPU= 80.567
  16:09:22 STEP  243 H=  60:45 +CPU= 81.021
  16:10:47 STEP  244 H=  61:00 +CPU= 83.556
  16:12:07 STEP  245 H=  61:15 +CPU= 78.572
  16:13:37 STEP  246 H=  61:30 +CPU= 87.784
  16:14:56 STEP  247 H=  61:45 +CPU= 77.913
  16:16:17 STEP  248 H=  62:00 +CPU= 79.266
  16:17:42 STEP  249 H=  62:15 +CPU= 84.285
  16:18:59 STEP  250 H=  62:30 +CPU= 76.015
  16:20:14 STEP  251 H=  62:45 +CPU= 73.942
  16:22:01 STEP  252 H=  63:00 +CPU=105.400
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000252
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000252
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000252
  16:23:24 STEP  253 H=  63:15 +CPU= 81.747
  16:24:47 STEP  254 H=  63:30 +CPU= 81.625
  16:26:08 STEP  255 H=  63:45 +CPU= 80.177
  16:27:28 STEP  256 H=  64:00 +CPU= 78.728
  16:28:51 STEP  257 H=  64:15 +CPU= 81.309
  16:30:11 STEP  258 H=  64:30 +CPU= 79.209
  16:31:27 STEP  259 H=  64:45 +CPU= 75.042
  16:32:52 STEP  260 H=  65:00 +CPU= 83.714
  16:34:11 STEP  261 H=  65:15 +CPU= 78.047
  16:35:27 STEP  262 H=  65:30 +CPU= 74.962
  16:36:51 STEP  263 H=  65:45 +CPU= 82.800
  16:38:34 STEP  264 H=  66:00 +CPU=100.688
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000264
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000264
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000264
  16:39:54 STEP  265 H=  66:15 +CPU= 76.882
  16:41:21 STEP  266 H=  66:30 +CPU= 78.391
  16:42:41 STEP  267 H=  66:45 +CPU= 78.149
  16:43:59 STEP  268 H=  67:00 +CPU= 77.568
  16:45:23 STEP  269 H=  67:15 +CPU= 82.783
  16:46:41 STEP  270 H=  67:30 +CPU= 76.617
  16:48:06 STEP  271 H=  67:45 +CPU= 84.712
  16:49:26 STEP  272 H=  68:00 +CPU= 78.695
  16:50:45 STEP  273 H=  68:15 +CPU= 77.710
  16:52:07 STEP  274 H=  68:30 +CPU= 81.117
  16:53:31 STEP  275 H=  68:45 +CPU= 82.389
  16:55:05 STEP  276 H=  69:00 +CPU= 92.840
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000276
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000276
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000276
  16:56:23 STEP  277 H=  69:15 +CPU= 76.974
  16:57:45 STEP  278 H=  69:30 +CPU= 80.237
  16:59:01 STEP  279 H=  69:45 +CPU= 72.247
  17:00:28 STEP  280 H=  70:00 +CPU= 85.267
  17:01:51 STEP  281 H=  70:15 +CPU= 81.081
  17:03:11 STEP  282 H=  70:30 +CPU= 77.988
  17:04:33 STEP  283 H=  70:45 +CPU= 81.005
  17:05:59 STEP  284 H=  71:00 +CPU= 84.853
  17:07:18 STEP  285 H=  71:15 +CPU= 77.907
  17:08:40 STEP  286 H=  71:30 +CPU= 80.531
  17:09:59 STEP  287 H=  71:45 +CPU= 76.582
  17:11:52 STEP  288 H=  72:00 +CPU=109.935
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000288
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000288
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000288
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000192
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000192
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000192
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000204
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000204
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000204
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000216
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000216
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000216
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000228
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000228
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000228
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000240
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000240
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000240
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000252
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000252
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000252
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000264
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000264
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000264
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000276
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000276
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000276
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0_r669128754_2.zip
Uploading the intermediate file: upload_file_2.zip
  17:13:15 STEP  289 H=  72:15 +CPU= 79.973
  17:14:36 STEP  290 H=  72:30 +CPU= 79.577
  17:15:54 STEP  291 H=  72:45 +CPU= 77.108
  17:17:14 STEP  292 H=  73:00 +CPU= 78.721
  17:18:39 STEP  293 H=  73:15 +CPU= 82.158
  17:20:06 STEP  294 H=  73:30 +CPU= 85.133
  17:21:28 STEP  295 H=  73:45 +CPU= 80.781
  17:22:54 STEP  296 H=  74:00 +CPU= 84.634
  17:24:17 STEP  297 H=  74:15 +CPU= 80.851
  17:25:40 STEP  298 H=  74:30 +CPU= 82.157
  17:26:59 STEP  299 H=  74:45 +CPU= 77.520
  17:28:43 STEP  300 H=  75:00 +CPU=101.742
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000300
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000300
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000300
  17:30:01 STEP  301 H=  75:15 +CPU= 77.642
  17:31:26 STEP  302 H=  75:30 +CPU= 83.308
  17:32:49 STEP  303 H=  75:45 +CPU= 81.600
  17:34:08 STEP  304 H=  76:00 +CPU= 78.283
  17:35:34 STEP  305 H=  76:15 +CPU= 84.556
  17:36:57 STEP  306 H=  76:30 +CPU= 81.678
  17:38:14 STEP  307 H=  76:45 +CPU= 75.735
  17:39:37 STEP  308 H=  77:00 +CPU= 81.712
  17:40:59 STEP  309 H=  77:15 +CPU= 80.960
  17:42:21 STEP  310 H=  77:30 +CPU= 81.140
  17:43:45 STEP  311 H=  77:45 +CPU= 82.915
  17:45:25 STEP  312 H=  78:00 +CPU= 97.650
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000312
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000312
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000312
  17:46:46 STEP  313 H=  78:15 +CPU= 80.660
  17:48:12 STEP  314 H=  78:30 +CPU= 85.061
  17:49:32 STEP  315 H=  78:45 +CPU= 78.879
  17:50:55 STEP  316 H=  79:00 +CPU= 81.872
  17:52:17 STEP  317 H=  79:15 +CPU= 80.824
  17:53:40 STEP  318 H=  79:30 +CPU= 81.788
  17:55:03 STEP  319 H=  79:45 +CPU= 82.164
  17:56:31 STEP  320 H=  80:00 +CPU= 86.935
  17:57:56 STEP  321 H=  80:15 +CPU= 83.838
  17:59:14 STEP  322 H=  80:30 +CPU= 76.516
  18:00:35 STEP  323 H=  80:45 +CPU= 79.419
  18:02:11 STEP  324 H=  81:00 +CPU= 94.191
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000324
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000324
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000324
  18:03:34 STEP  325 H=  81:15 +CPU= 81.870
  18:04:48 STEP  326 H=  81:30 +CPU= 73.140
  18:06:12 STEP  327 H=  81:45 +CPU= 81.989
  18:07:38 STEP  328 H=  82:00 +CPU= 85.651
  18:08:55 STEP  329 H=  82:15 +CPU= 75.693
  18:10:19 STEP  330 H=  82:30 +CPU= 82.621
  18:11:41 STEP  331 H=  82:45 +CPU= 80.141
  18:13:03 STEP  332 H=  83:00 +CPU= 80.513
  18:14:24 STEP  333 H=  83:15 +CPU= 79.619
  18:15:44 STEP  334 H=  83:30 +CPU= 78.666
  18:17:07 STEP  335 H=  83:45 +CPU= 81.845
  18:18:53 STEP  336 H=  84:00 +CPU=105.191
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000336
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000336
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000336
  18:20:17 STEP  337 H=  84:15 +CPU= 82.151
  18:21:38 STEP  338 H=  84:30 +CPU= 80.075
  18:22:57 STEP  339 H=  84:45 +CPU= 78.207
  18:24:19 STEP  340 H=  85:00 +CPU= 80.695
  18:25:45 STEP  341 H=  85:15 +CPU= 84.218
  18:27:07 STEP  342 H=  85:30 +CPU= 81.395
  18:28:24 STEP  343 H=  85:45 +CPU= 75.979
  18:29:41 STEP  344 H=  86:00 +CPU= 75.497
  18:31:00 STEP  345 H=  86:15 +CPU= 78.251
  18:32:14 STEP  346 H=  86:30 +CPU= 72.438
  18:33:27 STEP  347 H=  86:45 +CPU= 72.686
  18:35:00 STEP  348 H=  87:00 +CPU= 91.491
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000348
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000348
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000348
  18:36:18 STEP  349 H=  87:15 +CPU= 77.286
  18:37:33 STEP  350 H=  87:30 +CPU= 74.645
  18:38:50 STEP  351 H=  87:45 +CPU= 75.195
  18:40:05 STEP  352 H=  88:00 +CPU= 71.113
  18:41:24 STEP  353 H=  88:15 +CPU= 75.457
  18:42:40 STEP  354 H=  88:30 +CPU= 75.097
  18:43:55 STEP  355 H=  88:45 +CPU= 72.599
  18:45:16 STEP  356 H=  89:00 +CPU= 76.962
  18:46:34 STEP  357 H=  89:15 +CPU= 75.193
  18:47:47 STEP  358 H=  89:30 +CPU= 71.187
  18:49:00 STEP  359 H=  89:45 +CPU= 72.602
  18:50:33 STEP  360 H=  90:00 +CPU= 91.386
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000360
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000360
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000360
  18:51:48 STEP  361 H=  90:15 +CPU= 73.889
  18:53:06 STEP  362 H=  90:30 +CPU= 75.461
  18:54:20 STEP  363 H=  90:45 +CPU= 72.269
  18:55:41 STEP  364 H=  91:00 +CPU= 79.045
  18:56:58 STEP  365 H=  91:15 +CPU= 76.123
  18:58:15 STEP  366 H=  91:30 +CPU= 74.632
  18:59:33 STEP  367 H=  91:45 +CPU= 76.729
  19:00:54 STEP  368 H=  92:00 +CPU= 79.810
  19:02:14 STEP  369 H=  92:15 +CPU= 78.064
  19:03:30 STEP  370 H=  92:30 +CPU= 74.997
  19:04:48 STEP  371 H=  92:45 +CPU= 75.479
  19:06:15 STEP  372 H=  93:00 +CPU= 83.901
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000372
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000372
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000372
  19:07:26 STEP  373 H=  93:15 +CPU= 67.202
  19:08:37 STEP  374 H=  93:30 +CPU= 67.920
  19:09:45 STEP  375 H=  93:45 +CPU= 65.947
  19:10:57 STEP  376 H=  94:00 +CPU= 69.694
  19:12:10 STEP  377 H=  94:15 +CPU= 66.163
  19:13:23 STEP  378 H=  94:30 +CPU= 66.746
  19:14:37 STEP  379 H=  94:45 +CPU= 68.511
  19:15:51 STEP  380 H=  95:00 +CPU= 66.357
  19:16:59 STEP  381 H=  95:15 +CPU= 62.023
  19:18:10 STEP  382 H=  95:30 +CPU= 62.354
  19:19:25 STEP  383 H=  95:45 +CPU= 66.359
  19:21:05 STEP  384 H=  96:00 +CPU= 92.164
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000384
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000384
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000384
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000288
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000288
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000288
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000300
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000300
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000300
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000312
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000312
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000312
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000324
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000324
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000324
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000336
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000336
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000336
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000348
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000348
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000348
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000360
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000360
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000360
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000372
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000372
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000372
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0_r669128754_3.zip
Uploading the intermediate file: upload_file_3.zip
  19:22:16 STEP  385 H=  96:15 +CPU= 65.109
  19:23:29 STEP  386 H=  96:30 +CPU= 66.385
  19:24:45 STEP  387 H=  96:45 +CPU= 68.245
  19:25:56 STEP  388 H=  97:00 +CPU= 66.666
  19:27:08 STEP  389 H=  97:15 +CPU= 66.405
  19:28:24 STEP  390 H=  97:30 +CPU= 69.444
  19:29:45 STEP  391 H=  97:45 +CPU= 70.878
  19:31:08 STEP  392 H=  98:00 +CPU= 75.397
  19:32:25 STEP  393 H=  98:15 +CPU= 72.463
  19:33:43 STEP  394 H=  98:30 +CPU= 71.480
  19:35:05 STEP  395 H=  98:45 +CPU= 78.556
  19:36:37 STEP  396 H=  99:00 +CPU= 89.379
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000396
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000396
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000396
  19:37:53 STEP  397 H=  99:15 +CPU= 74.887
  19:39:12 STEP  398 H=  99:30 +CPU= 77.384
  19:40:30 STEP  399 H=  99:45 +CPU= 76.557
  19:41:49 STEP  400 H= 100:00 +CPU= 77.493
  19:43:04 STEP  401 H= 100:15 +CPU= 72.576
  19:44:20 STEP  402 H= 100:30 +CPU= 73.877
  19:45:40 STEP  403 H= 100:45 +CPU= 77.661
  19:47:00 STEP  404 H= 101:00 +CPU= 76.950
  19:48:11 STEP  405 H= 101:15 +CPU= 69.881
  19:49:31 STEP  406 H= 101:30 +CPU= 75.683
  19:50:51 STEP  407 H= 101:45 +CPU= 77.698
  19:52:24 STEP  408 H= 102:00 +CPU= 90.370
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000408
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000408
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000408
  19:53:45 STEP  409 H= 102:15 +CPU= 78.655
  19:55:03 STEP  410 H= 102:30 +CPU= 76.446
  19:56:21 STEP  411 H= 102:45 +CPU= 74.342
  19:57:39 STEP  412 H= 103:00 +CPU= 75.595
  19:58:56 STEP  413 H= 103:15 +CPU= 73.180
  20:00:11 STEP  414 H= 103:30 +CPU= 72.289
  20:01:21 STEP  415 H= 103:45 +CPU= 67.336
  20:02:32 STEP  416 H= 104:00 +CPU= 68.935
  20:03:46 STEP  417 H= 104:15 +CPU= 72.031
  20:04:56 STEP  418 H= 104:30 +CPU= 68.942
  20:06:07 STEP  419 H= 104:45 +CPU= 68.658
  20:07:34 STEP  420 H= 105:00 +CPU= 85.687
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000420
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000420
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000420
  20:08:46 STEP  421 H= 105:15 +CPU= 70.197
  20:09:56 STEP  422 H= 105:30 +CPU= 67.795
  20:11:07 STEP  423 H= 105:45 +CPU= 69.376
  20:12:21 STEP  424 H= 106:00 +CPU= 71.688
  20:13:33 STEP  425 H= 106:15 +CPU= 70.009
  20:14:42 STEP  426 H= 106:30 +CPU= 68.233
  20:15:58 STEP  427 H= 106:45 +CPU= 73.311
  20:17:10 STEP  428 H= 107:00 +CPU= 70.108
  20:18:20 STEP  429 H= 107:15 +CPU= 68.522
  20:19:31 STEP  430 H= 107:30 +CPU= 68.886
  20:20:40 STEP  431 H= 107:45 +CPU= 67.424
  20:22:15 STEP  432 H= 108:00 +CPU= 92.504
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000432
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000432
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000432
  20:23:24 STEP  433 H= 108:15 +CPU= 67.677
  20:24:34 STEP  434 H= 108:30 +CPU= 68.410
  20:25:44 STEP  435 H= 108:45 +CPU= 67.829
  20:26:58 STEP  436 H= 109:00 +CPU= 71.516
  20:28:07 STEP  437 H= 109:15 +CPU= 67.705
  20:29:19 STEP  438 H= 109:30 +CPU= 69.907
  20:30:30 STEP  439 H= 109:45 +CPU= 68.491
  20:31:41 STEP  440 H= 110:00 +CPU= 69.771
  20:32:54 STEP  441 H= 110:15 +CPU= 70.481
  20:34:05 STEP  442 H= 110:30 +CPU= 68.697
  20:35:15 STEP  443 H= 110:45 +CPU= 69.104
  20:36:44 STEP  444 H= 111:00 +CPU= 86.723
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000444
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000444
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000444
  20:37:53 STEP  445 H= 111:15 +CPU= 67.013
  20:39:05 STEP  446 H= 111:30 +CPU= 69.990
  20:40:18 STEP  447 H= 111:45 +CPU= 71.233
  20:41:28 STEP  448 H= 112:00 +CPU= 67.494
  20:42:38 STEP  449 H= 112:15 +CPU= 69.106
  20:43:50 STEP  450 H= 112:30 +CPU= 69.457
  20:44:55 STEP  451 H= 112:45 +CPU= 64.116
  20:46:07 STEP  452 H= 113:00 +CPU= 69.520
  20:47:18 STEP  453 H= 113:15 +CPU= 69.378
  20:48:28 STEP  454 H= 113:30 +CPU= 67.982
  20:49:40 STEP  455 H= 113:45 +CPU= 70.417
  20:51:07 STEP  456 H= 114:00 +CPU= 84.436
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000456
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000456
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000456
  20:52:18 STEP  457 H= 114:15 +CPU= 69.179
  20:53:27 STEP  458 H= 114:30 +CPU= 67.212
  20:54:39 STEP  459 H= 114:45 +CPU= 70.244
  20:55:50 STEP  460 H= 115:00 +CPU= 69.535
  20:57:03 STEP  461 H= 115:15 +CPU= 70.939
  20:58:13 STEP  462 H= 115:30 +CPU= 68.484
  20:59:24 STEP  463 H= 115:45 +CPU= 68.390
  21:00:39 STEP  464 H= 116:00 +CPU= 70.468
  21:01:47 STEP  465 H= 116:15 +CPU= 67.003
  21:02:59 STEP  466 H= 116:30 +CPU= 69.911
  21:04:11 STEP  467 H= 116:45 +CPU= 69.980
  21:05:36 STEP  468 H= 117:00 +CPU= 83.937
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000468
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000468
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000468
  21:06:51 STEP  469 H= 117:15 +CPU= 73.653
  21:08:02 STEP  470 H= 117:30 +CPU= 70.597
  21:09:13 STEP  471 H= 117:45 +CPU= 70.199
  21:10:27 STEP  472 H= 118:00 +CPU= 72.360
  21:11:36 STEP  473 H= 118:15 +CPU= 67.421
  21:12:49 STEP  474 H= 118:30 +CPU= 71.839
  21:14:04 STEP  475 H= 118:45 +CPU= 74.291
  21:15:14 STEP  476 H= 119:00 +CPU= 69.077
  21:16:23 STEP  477 H= 119:15 +CPU= 66.348
  21:17:36 STEP  478 H= 119:30 +CPU= 71.388
  21:18:42 STEP  479 H= 119:45 +CPU= 65.464
  21:20:15 STEP  480 H= 120:00 +CPU= 91.013
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000480
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000480
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000480
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000384
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000384
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000384
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000396
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000396
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000396
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000408
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000408
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000408
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000420
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000420
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000420
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000432
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000432
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000432
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000444
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000444
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000444
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000456
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000456
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000456
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000468
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000468
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000468
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0_r669128754_4.zip
Uploading the intermediate file: upload_file_4.zip
  21:21:26 STEP  481 H= 120:15 +CPU= 69.343
  21:22:37 STEP  482 H= 120:30 +CPU= 70.020
  21:23:48 STEP  483 H= 120:45 +CPU= 69.972
  21:25:03 STEP  484 H= 121:00 +CPU= 73.773
  21:26:11 STEP  485 H= 121:15 +CPU= 67.238
  21:27:27 STEP  486 H= 121:30 +CPU= 75.979
  21:28:46 STEP  487 H= 121:45 +CPU= 77.985
  21:29:55 STEP  488 H= 122:00 +CPU= 68.233
  21:31:07 STEP  489 H= 122:15 +CPU= 68.979
  21:32:16 STEP  490 H= 122:30 +CPU= 68.097
  21:33:22 STEP  491 H= 122:45 +CPU= 64.461
  21:34:43 STEP  492 H= 123:00 +CPU= 80.268
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000492
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000492
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000492
  21:35:54 STEP  493 H= 123:15 +CPU= 69.977
  21:37:01 STEP  494 H= 123:30 +CPU= 66.664
  21:38:13 STEP  495 H= 123:45 +CPU= 70.731
  21:39:24 STEP  496 H= 124:00 +CPU= 70.090
  21:40:34 STEP  497 H= 124:15 +CPU= 68.831
  21:41:41 STEP  498 H= 124:30 +CPU= 67.243
  21:42:51 STEP  499 H= 124:45 +CPU= 68.710
  21:44:05 STEP  500 H= 125:00 +CPU= 73.234
  21:45:19 STEP  501 H= 125:15 +CPU= 73.204
  21:46:29 STEP  502 H= 125:30 +CPU= 68.878
  21:47:36 STEP  503 H= 125:45 +CPU= 66.972
  21:49:04 STEP  504 H= 126:00 +CPU= 86.751
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000504
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000504
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000504
  21:50:15 STEP  505 H= 126:15 +CPU= 69.998
  21:51:28 STEP  506 H= 126:30 +CPU= 72.803
  21:52:37 STEP  507 H= 126:45 +CPU= 67.744
  21:53:48 STEP  508 H= 127:00 +CPU= 70.515
  21:54:56 STEP  509 H= 127:15 +CPU= 66.852
  21:56:03 STEP  510 H= 127:30 +CPU= 67.095
  21:57:15 STEP  511 H= 127:45 +CPU= 70.832
  21:58:28 STEP  512 H= 128:00 +CPU= 71.296
  21:59:41 STEP  513 H= 128:15 +CPU= 73.065
  22:00:54 STEP  514 H= 128:30 +CPU= 71.899
  22:02:06 STEP  515 H= 128:45 +CPU= 70.503
  22:03:26 STEP  516 H= 129:00 +CPU= 79.051
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000516
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000516
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000516
  22:04:36 STEP  517 H= 129:15 +CPU= 69.101
  22:05:45 STEP  518 H= 129:30 +CPU= 68.874
  22:06:54 STEP  519 H= 129:45 +CPU= 67.641
  22:08:19 STEP  520 H= 130:00 +CPU= 60.436
  22:09:09 STEP  521 H= 130:15 +CPU= 48.649
  22:10:06 STEP  522 H= 130:30 +CPU= 56.761
  22:11:05 STEP  523 H= 130:45 +CPU= 57.030
  22:12:07 STEP  524 H= 131:00 +CPU= 61.850
  22:13:08 STEP  525 H= 131:15 +CPU= 59.747
  22:14:09 STEP  526 H= 131:30 +CPU= 59.337
  22:15:19 STEP  527 H= 131:45 +CPU= 67.159
  22:16:50 STEP  528 H= 132:00 +CPU= 87.967
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000528
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000528
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000528
  22:18:00 STEP  529 H= 132:15 +CPU= 67.077
  22:19:13 STEP  530 H= 132:30 +CPU= 69.508
  22:20:19 STEP  531 H= 132:45 +CPU= 63.548
  22:21:25 STEP  532 H= 133:00 +CPU= 64.338
  22:22:35 STEP  533 H= 133:15 +CPU= 67.921
  22:23:42 STEP  534 H= 133:30 +CPU= 64.356
  22:24:51 STEP  535 H= 133:45 +CPU= 66.752
  22:26:01 STEP  536 H= 134:00 +CPU= 67.571
  22:27:08 STEP  537 H= 134:15 +CPU= 63.167
  22:28:23 STEP  538 H= 134:30 +CPU= 62.565
  22:29:35 STEP  539 H= 134:45 +CPU= 67.553
  22:30:57 STEP  540 H= 135:00 +CPU= 82.086
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000540
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000540
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000540
  22:32:05 STEP  541 H= 135:15 +CPU= 66.994
  22:33:16 STEP  542 H= 135:30 +CPU= 69.975
  22:34:25 STEP  543 H= 135:45 +CPU= 67.900
  22:35:34 STEP  544 H= 136:00 +CPU= 67.972
  22:36:41 STEP  545 H= 136:15 +CPU= 66.507
  22:37:52 STEP  546 H= 136:30 +CPU= 70.344
  22:39:05 STEP  547 H= 136:45 +CPU= 71.690
  22:40:13 STEP  548 H= 137:00 +CPU= 66.641
  22:41:22 STEP  549 H= 137:15 +CPU= 68.919
  22:42:33 STEP  550 H= 137:30 +CPU= 69.943
  22:43:41 STEP  551 H= 137:45 +CPU= 66.299
  22:45:07 STEP  552 H= 138:00 +CPU= 85.495
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000552
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000552
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000552
  22:46:13 STEP  553 H= 138:15 +CPU= 65.508
  22:47:26 STEP  554 H= 138:30 +CPU= 71.728
  22:48:38 STEP  555 H= 138:45 +CPU= 70.576
  22:49:49 STEP  556 H= 139:00 +CPU= 70.034
  22:50:59 STEP  557 H= 139:15 +CPU= 68.904
  22:52:10 STEP  558 H= 139:30 +CPU= 69.035
  22:53:17 STEP  559 H= 139:45 +CPU= 66.748
  22:54:30 STEP  560 H= 140:00 +CPU= 71.506
  22:55:37 STEP  561 H= 140:15 +CPU= 66.911
  22:56:44 STEP  562 H= 140:30 +CPU= 65.834
  22:57:49 STEP  563 H= 140:45 +CPU= 64.193
  22:59:18 STEP  564 H= 141:00 +CPU= 88.268
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000564
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000564
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000564
  23:00:26 STEP  565 H= 141:15 +CPU= 67.430
  23:01:35 STEP  566 H= 141:30 +CPU= 68.031
  23:02:44 STEP  567 H= 141:45 +CPU= 67.939
  23:03:50 STEP  568 H= 142:00 +CPU= 65.538
  23:05:03 STEP  569 H= 142:15 +CPU= 71.902
  23:06:15 STEP  570 H= 142:30 +CPU= 70.127
  23:07:25 STEP  571 H= 142:45 +CPU= 68.660
  23:08:40 STEP  572 H= 143:00 +CPU= 74.131
  23:09:52 STEP  573 H= 143:15 +CPU= 70.702
  23:11:07 STEP  574 H= 143:30 +CPU= 72.605
  23:12:20 STEP  575 H= 143:45 +CPU= 70.212
  23:14:00 STEP  576 H= 144:00 +CPU= 98.391
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000576
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000576
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000576
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000480
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000480
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000480
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000492
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000492
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000492
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000504
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000504
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000504
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000516
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000516
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000516
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000528
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000528
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000528
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000540
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000540
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000540
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000552
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000552
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000552
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMGGh7zg+000564
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMSHh7zg+000564
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12208976/ICMUAh7zg+000564
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a1ot_2016092300_15_991_12208976_0_r669128754_5.zip
Uploading the intermediate file: upload_file_5.zip
  23:15:16 STEP  577 H= 144:15 +CPU= 73.324
  23:16:33 STEP  578 H= 144:30 +CPU= 75.508
  23:17:47 STEP  579 H= 144:45 +CPU= 73.231
  23:19:01 STEP  580 H= 145:00 +CPU= 73.419
  23:20:14 STEP  581 H= 145:15 +CPU= 71.544
  23:21:28 STEP  582 H= 145:30 +CPU= 72.549
  23:22:44 STEP  583 H= 145:45 +CPU= 75.364
  23:23:58 STEP  584 H= 146:00 +CPU= 72.226
  23:25:08 STEP  585 H= 146:15 +CPU= 69.405
  23:26:24 STEP  586 H= 146:30 +CPU= 74.654
  23:27:37 STEP  587 H= 146:45 +CPU= 72.174
  23:29:04 STEP  588 H= 147:00 +CPU= 85.998
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000588
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000588
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000588
  23:30:19 STEP  589 H= 147:15 +CPU= 73.904
  23:31:31 STEP  590 H= 147:30 +CPU= 71.243
  23:32:48 STEP  591 H= 147:45 +CPU= 76.697
  23:34:02 STEP  592 H= 148:00 +CPU= 72.589
  23:35:16 STEP  593 H= 148:15 +CPU= 73.586
  23:36:27 STEP  594 H= 148:30 +CPU= 69.731
  23:37:42 STEP  595 H= 148:45 +CPU= 73.316
  23:38:56 STEP  596 H= 149:00 +CPU= 73.197
  23:40:12 STEP  597 H= 149:15 +CPU= 75.543
  23:41:28 STEP  598 H= 149:30 +CPU= 74.313
  23:42:40 STEP  599 H= 149:45 +CPU= 71.357
  23:44:13 STEP  600 H= 150:00 +CPU= 91.500
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000600
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000600
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000600
  23:45:27 STEP  601 H= 150:15 +CPU= 71.086
  23:46:36 STEP  602 H= 150:30 +CPU= 68.588
  23:47:51 STEP  603 H= 150:45 +CPU= 73.203
  23:49:03 STEP  604 H= 151:00 +CPU= 71.645
  23:50:13 STEP  605 H= 151:15 +CPU= 68.774
  23:51:26 STEP  606 H= 151:30 +CPU= 72.207
  23:52:28 STEP  607 H= 151:45 +CPU= 61.411
  23:53:34 STEP  608 H= 152:00 +CPU= 65.605
  23:54:42 STEP  609 H= 152:15 +CPU= 67.300
  23:55:51 STEP  610 H= 152:30 +CPU= 67.745
  23:57:00 STEP  611 H= 152:45 +CPU= 68.075
  23:58:28 STEP  612 H= 153:00 +CPU= 87.153
Moving to projects directory: /home/steve/slots/18/ICMGGh7zg+000612
Moving to projects directory: /home/steve/slots/18/ICMSHh7zg+000612
Moving to projects directory: /home/steve/slots/18/ICMUAh7zg+000612
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01

 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
         630  9.999999999999998E-005
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.112942252861804E-02 0.999999999999998E-08 0.106111919444244E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.532926271906845E-06 0.000000000000000E+00 0.235769542003753E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.549663327914405E-07 0.000000000000000E+00 0.409782644545962E-04
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.248638711031289E-07 0.000000000000000E+00 0.941478063433477E-04
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.486997462584753E-08 0.000000000000000E+00 0.940165502749611E-05
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.145973252922963E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.533569231861014E-06 0.000000000000000E+00 0.237865717027892E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.549441750215672E-07 0.000000000000000E+00 0.516641724698275E-04
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.145933977632731E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.221093216148668E-07 0.000000000000000E+00 0.449008648426110E-04
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.240706343219099E-07 0.000000000000000E+00 0.114979320887227E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.289671091951212E-06 0.000000000000000E+00 0.200787912621304E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.112942220196761E-06 0.000000000000000E+00 0.194465629814909E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.340015110248924E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =   612 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   612 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000612 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+000612
 MPI-TASK:   1 -    303292266 BYTES IN      6 RECORDS TRANSFERRED IN    0.0018 SECONDS168495.7 Mbytes/s, TOTAL TIME=    0.0466( 3.9%)
 NSTEP =   612 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+000612 MODE=a
 IO-STREAM CLOSED - ./ICMSHh7zg+000612
 MPI-TASK:   1 -    707765250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0297 SECONDS 23830.5 Mbytes/s, TOTAL TIME=    0.7483( 4.0%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000612 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000612
 MPI-TASK:   1 -   4852676247 BYTES IN    125 RECORDS TRANSFERRED IN    0.0172 SECONDS282132.3 Mbytes/s, TOTAL TIME=    0.4891( 3.5%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000612 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000612
 MPI-TASK:   1 -    294718633 BYTES IN      2 RECORDS TRANSFERRED IN    0.0003 SECONDS982395.5 Mbytes/s, TOTAL TIME=    0.0113( 2.7%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000612 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000612
 MPI-TASK:   1 -    303292266 BYTES IN      4 RECORDS TRANSFERRED IN    0.0006 SECONDS505487.1 Mbytes/s, TOTAL TIME=    0.0182( 3.3%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =   612 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   612 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000612 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+000612
 MPI-TASK:   1 -    303292257 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS433274.7 Mbytes/s, TOTAL TIME=    0.0183( 3.8%)
 23:58:28 STEP  612 H= 153:00 +CPU= 87.153
 NSTEP =   613 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 23:57:00 A00000000 STEPO      612    68.352   68.352   69.582    808:10  -2752:** 0.28482202874385E-05       2GB       0MB
 23:57:00 0AA000000 STEPO      612     0.087    0.087    0.099    808:10  -2752:** 0.28482202874385E-05       2GB       0MB
 23:57:06 FULLPOS-B DYNFPOS    612     5.261    5.261    5.334    808:15  -2752:** 0.28482202874385E-05       2GB       0MB
 23:57:15 FULLPOS-V DYNFPOS    612     9.473    9.473    9.602    808:24  -2752:** 0.28482202874385E-05       2GB       0MB
 23:57:17 FULLPOS-T DYNFPOS    612     1.195    1.195    1.199    808:26  -2752:** 0.28482202874385E-05       2GB       0MB
 23:57:17 0AAA00AAA STEPO      612     0.834    0.834    0.836    808:26  -2752:** 0.28482202874385E-05       2GB       0MB
 23:58:28 0AAA00AAA STEPO      613    70.029   70.029   70.995    809:36  -2751:** 0.28482202874385E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     576",
 CTIME   = "00060000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         564, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /home/steve/slots/18/progress_file_12208976.xml

<running_values>
  <last_cpu_time>48603.050000</last_cpu_time>
  <upload_file_number>6</upload_file_number>
  <last_iter>613</last_iter>
  <last_upload>518400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 cpdn.org