climateprediction.net home page
Task 22446052

Task 22446052

Name oifs_43r3_bl_a36f_2016092300_20_1018_12292446_0
Workunit 12292446
Created 13 Jun 2024, 1:59:31 UTC
Sent 13 Jun 2024, 22:27:15 UTC
Report deadline 12 Aug 2024, 22:27:15 UTC
Received 17 Jun 2024, 7:28:59 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1536980
Run time 1 hours 9 min 41 sec
CPU time 1 hours 8 min 30 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 9.95 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,509.72 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,410.08 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
=  7.338
  02:17:38 STEP  131 H=  32:45 +CPU=  7.544
  02:17:48 STEP  132 H=  33:00 +CPU= 10.029
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000132
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000132
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000132
  02:17:56 STEP  133 H=  33:15 +CPU=  7.598
  02:18:03 STEP  134 H=  33:30 +CPU=  7.446
  02:18:11 STEP  135 H=  33:45 +CPU=  7.439
  02:18:19 STEP  136 H=  34:00 +CPU=  7.808
  02:18:26 STEP  137 H=  34:15 +CPU=  7.415
  02:18:33 STEP  138 H=  34:30 +CPU=  7.408
  02:18:41 STEP  139 H=  34:45 +CPU=  7.417
  02:18:48 STEP  140 H=  35:00 +CPU=  7.441
  02:18:55 STEP  141 H=  35:15 +CPU=  7.139
  02:19:03 STEP  142 H=  35:30 +CPU=  7.380
  02:19:10 STEP  143 H=  35:45 +CPU=  7.183
  02:19:21 STEP  144 H=  36:00 +CPU= 10.641
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000144
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000144
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000144
  02:19:28 STEP  145 H=  36:15 +CPU=  7.340
  02:19:35 STEP  146 H=  36:30 +CPU=  7.313
  02:19:43 STEP  147 H=  36:45 +CPU=  7.156
  02:19:50 STEP  148 H=  37:00 +CPU=  7.328
  02:19:57 STEP  149 H=  37:15 +CPU=  7.193
  02:20:04 STEP  150 H=  37:30 +CPU=  7.019
  02:20:12 STEP  151 H=  37:45 +CPU=  7.470
  02:20:19 STEP  152 H=  38:00 +CPU=  7.723
  02:20:27 STEP  153 H=  38:15 +CPU=  7.243
  02:20:34 STEP  154 H=  38:30 +CPU=  7.190
  02:20:41 STEP  155 H=  38:45 +CPU=  7.330
  02:20:51 STEP  156 H=  39:00 +CPU=  9.920
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000156
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000156
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000156
  02:20:59 STEP  157 H=  39:15 +CPU=  7.529
  02:21:06 STEP  158 H=  39:30 +CPU=  7.231
  02:21:13 STEP  159 H=  39:45 +CPU=  7.126
  02:21:21 STEP  160 H=  40:00 +CPU=  7.672
  02:21:28 STEP  161 H=  40:15 +CPU=  7.395
  02:21:35 STEP  162 H=  40:30 +CPU=  7.244
  02:21:42 STEP  163 H=  40:45 +CPU=  7.089
  02:21:50 STEP  164 H=  41:00 +CPU=  7.436
  02:21:57 STEP  165 H=  41:15 +CPU=  7.545
  02:22:05 STEP  166 H=  41:30 +CPU=  7.754
  02:22:13 STEP  167 H=  41:45 +CPU=  7.470
  02:22:23 STEP  168 H=  42:00 +CPU=  9.980
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000168
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000168
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000168
  02:22:30 STEP  169 H=  42:15 +CPU=  7.318
  02:22:37 STEP  170 H=  42:30 +CPU=  7.338
  02:22:45 STEP  171 H=  42:45 +CPU=  7.238
  02:22:52 STEP  172 H=  43:00 +CPU=  7.660
  02:23:00 STEP  173 H=  43:15 +CPU=  7.289
  02:23:07 STEP  174 H=  43:30 +CPU=  7.771
  02:23:15 STEP  175 H=  43:45 +CPU=  7.723
  02:23:23 STEP  176 H=  44:00 +CPU=  7.842
  02:23:30 STEP  177 H=  44:15 +CPU=  7.203
  02:23:37 STEP  178 H=  44:30 +CPU=  7.169
  02:23:45 STEP  179 H=  44:45 +CPU=  7.576
  02:23:54 STEP  180 H=  45:00 +CPU=  9.601
  02:24:02 STEP  181 H=  45:15 +CPU=  7.303
  02:24:09 STEP  182 H=  45:30 +CPU=  7.609
  02:24:17 STEP  183 H=  45:45 +CPU=  7.734
  02:24:25 STEP  184 H=  46:00 +CPU=  7.600
  02:24:32 STEP  185 H=  46:15 +CPU=  7.616
  02:24:40 STEP  186 H=  46:30 +CPU=  7.825
  02:24:48 STEP  187 H=  46:45 +CPU=  7.296
  02:24:56 STEP  188 H=  47:00 +CPU=  8.041
  02:25:03 STEP  189 H=  47:15 +CPU=  7.596
  02:25:11 STEP  190 H=  47:30 +CPU=  7.675
  02:25:18 STEP  191 H=  47:45 +CPU=  7.471
  02:25:29 STEP  192 H=  48:00 +CPU= 10.206
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000192
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000192
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000192
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000096
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000096
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000096
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000108
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000108
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000108
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000120
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000120
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000120
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000132
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000132
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000132
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000144
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000144
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000144
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000156
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000156
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000156
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000168
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000168
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000168
Zipping up the intermediate file: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_a36f_2016092300_20_1018_12292446_0_r424306766_1.zip
  02:25:36 STEP  193 H=  48:15 +CPU=  7.493
Uploading the intermediate file: upload_file_1.zip
  02:25:44 STEP  194 H=  48:30 +CPU=  7.627
  02:25:51 STEP  195 H=  48:45 +CPU=  7.588
  02:25:59 STEP  196 H=  49:00 +CPU=  7.555
  02:26:06 STEP  197 H=  49:15 +CPU=  7.181
  02:26:13 STEP  198 H=  49:30 +CPU=  7.218
  02:26:21 STEP  199 H=  49:45 +CPU=  7.731
  02:26:29 STEP  200 H=  50:00 +CPU=  7.620
  02:26:36 STEP  201 H=  50:15 +CPU=  7.204
  02:26:43 STEP  202 H=  50:30 +CPU=  7.534
  02:26:51 STEP  203 H=  50:45 +CPU=  7.628
  02:27:01 STEP  204 H=  51:00 +CPU= 10.087
  02:27:09 STEP  205 H=  51:15 +CPU=  7.623
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000204
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000204
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000204
  02:27:16 STEP  206 H=  51:30 +CPU=  7.630
  02:27:24 STEP  207 H=  51:45 +CPU=  7.662
  02:27:32 STEP  208 H=  52:00 +CPU=  7.864
  02:27:40 STEP  209 H=  52:15 +CPU=  7.728
  02:27:47 STEP  210 H=  52:30 +CPU=  7.649
  02:27:55 STEP  211 H=  52:45 +CPU=  7.711
  02:28:03 STEP  212 H=  53:00 +CPU=  7.901
  02:28:11 STEP  213 H=  53:15 +CPU=  7.624
  02:28:18 STEP  214 H=  53:30 +CPU=  7.770
  02:28:26 STEP  215 H=  53:45 +CPU=  7.368
  02:28:36 STEP  216 H=  54:00 +CPU=  9.692
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000216
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000216
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000216
  02:28:43 STEP  217 H=  54:15 +CPU=  7.565
  02:28:51 STEP  218 H=  54:30 +CPU=  7.604
  02:28:58 STEP  219 H=  54:45 +CPU=  7.703
  02:29:06 STEP  220 H=  55:00 +CPU=  7.624
  02:29:13 STEP  221 H=  55:15 +CPU=  7.337
  02:29:21 STEP  222 H=  55:30 +CPU=  7.810
  02:29:29 STEP  223 H=  55:45 +CPU=  8.063
  02:29:38 STEP  224 H=  56:00 +CPU=  8.146
  02:29:45 STEP  225 H=  56:15 +CPU=  7.578
  02:29:53 STEP  226 H=  56:30 +CPU=  7.436
  02:30:00 STEP  227 H=  56:45 +CPU=  7.673
  02:30:10 STEP  228 H=  57:00 +CPU=  9.775
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000228
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000228
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000228
  02:30:18 STEP  229 H=  57:15 +CPU=  7.497
  02:30:25 STEP  230 H=  57:30 +CPU=  7.404
  02:30:32 STEP  231 H=  57:45 +CPU=  7.294
  02:30:40 STEP  232 H=  58:00 +CPU=  8.040
  02:30:48 STEP  233 H=  58:15 +CPU=  7.787
  02:30:56 STEP  234 H=  58:30 +CPU=  7.964
  02:31:03 STEP  235 H=  58:45 +CPU=  7.310
  02:31:11 STEP  236 H=  59:00 +CPU=  7.660
  02:31:18 STEP  237 H=  59:15 +CPU=  7.406
  02:31:26 STEP  238 H=  59:30 +CPU=  7.540
  02:31:34 STEP  239 H=  59:45 +CPU=  7.512
  02:31:44 STEP  240 H=  60:00 +CPU= 10.348
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000240
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000240
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000240
  02:31:52 STEP  241 H=  60:15 +CPU=  7.565
  02:31:59 STEP  242 H=  60:30 +CPU=  7.582
  02:32:07 STEP  243 H=  60:45 +CPU=  7.420
  02:32:15 STEP  244 H=  61:00 +CPU=  8.153
  02:32:22 STEP  245 H=  61:15 +CPU=  7.744
  02:32:30 STEP  246 H=  61:30 +CPU=  7.626
  02:32:38 STEP  247 H=  61:45 +CPU=  7.671
  02:32:46 STEP  248 H=  62:00 +CPU=  7.802
  02:32:53 STEP  249 H=  62:15 +CPU=  7.737
  02:33:01 STEP  250 H=  62:30 +CPU=  7.813
  02:33:09 STEP  251 H=  62:45 +CPU=  7.680
  02:33:19 STEP  252 H=  63:00 +CPU= 10.136
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000252
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000252
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000252
  02:33:27 STEP  253 H=  63:15 +CPU=  7.563
  02:33:34 STEP  254 H=  63:30 +CPU=  7.495
  02:33:41 STEP  255 H=  63:45 +CPU=  7.238
  02:33:49 STEP  256 H=  64:00 +CPU=  7.676
  02:33:57 STEP  257 H=  64:15 +CPU=  7.628
  02:34:04 STEP  258 H=  64:30 +CPU=  7.734
  02:34:12 STEP  259 H=  64:45 +CPU=  7.829
  02:34:20 STEP  260 H=  65:00 +CPU=  8.115
  02:34:28 STEP  261 H=  65:15 +CPU=  7.564
  02:34:35 STEP  262 H=  65:30 +CPU=  7.406
  02:34:43 STEP  263 H=  65:45 +CPU=  7.563
  02:34:53 STEP  264 H=  66:00 +CPU=  9.980
  02:35:01 STEP  265 H=  66:15 +CPU=  7.706
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000264
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000264
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000264
  02:35:08 STEP  266 H=  66:30 +CPU=  7.561
  02:35:16 STEP  267 H=  66:45 +CPU=  7.763
  02:35:24 STEP  268 H=  67:00 +CPU=  7.946
  02:35:31 STEP  269 H=  67:15 +CPU=  7.462
  02:35:39 STEP  270 H=  67:30 +CPU=  7.491
  02:35:47 STEP  271 H=  67:45 +CPU=  7.696
  02:35:55 STEP  272 H=  68:00 +CPU=  8.075
  02:36:02 STEP  273 H=  68:15 +CPU=  7.693
  02:36:10 STEP  274 H=  68:30 +CPU=  7.685
  02:36:18 STEP  275 H=  68:45 +CPU=  7.732
  02:36:28 STEP  276 H=  69:00 +CPU=  9.887
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000276
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000276
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000276
  02:36:35 STEP  277 H=  69:15 +CPU=  7.608
  02:36:43 STEP  278 H=  69:30 +CPU=  7.584
  02:36:51 STEP  279 H=  69:45 +CPU=  7.652
  02:36:59 STEP  280 H=  70:00 +CPU=  8.137
  02:37:07 STEP  281 H=  70:15 +CPU=  7.696
  02:37:14 STEP  282 H=  70:30 +CPU=  7.275
  02:37:22 STEP  283 H=  70:45 +CPU=  7.690
  02:37:30 STEP  284 H=  71:00 +CPU=  8.323
  02:37:37 STEP  285 H=  71:15 +CPU=  7.274
  02:37:45 STEP  286 H=  71:30 +CPU=  7.587
  02:37:52 STEP  287 H=  71:45 +CPU=  7.387
  02:38:04 STEP  288 H=  72:00 +CPU= 11.369
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000288
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000288
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000288
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000192
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000192
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000192
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000204
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000204
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000204
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000216
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000216
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000216
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000228
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000228
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000228
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000240
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000240
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000240
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000252
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000252
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000252
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000264
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000264
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000264
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000276
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000276
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000276
Zipping up the intermediate file: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_a36f_2016092300_20_1018_12292446_0_r424306766_2.zip
  02:38:11 STEP  289 H=  72:15 +CPU=  7.938
Uploading the intermediate file: upload_file_2.zip
  02:38:19 STEP  290 H=  72:30 +CPU=  7.496
  02:38:27 STEP  291 H=  72:45 +CPU=  7.714
  02:38:35 STEP  292 H=  73:00 +CPU=  8.024
  02:38:42 STEP  293 H=  73:15 +CPU=  7.505
  02:38:50 STEP  294 H=  73:30 +CPU=  7.636
  02:38:58 STEP  295 H=  73:45 +CPU=  7.727
  02:39:06 STEP  296 H=  74:00 +CPU=  7.994
  02:39:13 STEP  297 H=  74:15 +CPU=  7.723
  02:39:21 STEP  298 H=  74:30 +CPU=  7.983
  02:39:30 STEP  299 H=  74:45 +CPU=  8.189
  02:39:40 STEP  300 H=  75:00 +CPU= 10.427
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000300
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000300
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000300
  02:39:48 STEP  301 H=  75:15 +CPU=  7.659
  02:39:56 STEP  302 H=  75:30 +CPU=  7.870
  02:40:04 STEP  303 H=  75:45 +CPU=  7.913
  02:40:12 STEP  304 H=  76:00 +CPU=  8.421
  02:40:19 STEP  305 H=  76:15 +CPU=  7.390
  02:40:27 STEP  306 H=  76:30 +CPU=  7.274
  02:40:35 STEP  307 H=  76:45 +CPU=  7.913
  02:40:43 STEP  308 H=  77:00 +CPU=  8.265
  02:40:51 STEP  309 H=  77:15 +CPU=  7.863
  02:40:59 STEP  310 H=  77:30 +CPU=  7.891
  02:41:06 STEP  311 H=  77:45 +CPU=  7.706
  02:41:16 STEP  312 H=  78:00 +CPU=  9.878
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000312
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000312
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000312
  02:41:24 STEP  313 H=  78:15 +CPU=  7.468
  02:41:31 STEP  314 H=  78:30 +CPU=  7.380
  02:41:38 STEP  315 H=  78:45 +CPU=  7.299
  02:41:46 STEP  316 H=  79:00 +CPU=  7.920
  02:41:54 STEP  317 H=  79:15 +CPU=  7.452
  02:42:02 STEP  318 H=  79:30 +CPU=  7.867
  02:42:09 STEP  319 H=  79:45 +CPU=  7.753
  02:42:18 STEP  320 H=  80:00 +CPU=  8.283
  02:42:26 STEP  321 H=  80:15 +CPU=  8.028
  02:42:33 STEP  322 H=  80:30 +CPU=  7.635
  02:42:41 STEP  323 H=  80:45 +CPU=  7.523
  02:42:51 STEP  324 H=  81:00 +CPU= 10.211
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000324
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000324
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000324
  02:42:59 STEP  325 H=  81:15 +CPU=  8.123
  02:43:07 STEP  326 H=  81:30 +CPU=  7.613
  02:43:15 STEP  327 H=  81:45 +CPU=  7.672
  02:43:23 STEP  328 H=  82:00 +CPU=  8.072
  02:43:30 STEP  329 H=  82:15 +CPU=  7.493
  02:43:38 STEP  330 H=  82:30 +CPU=  7.866
  02:43:46 STEP  331 H=  82:45 +CPU=  7.505
  02:43:54 STEP  332 H=  83:00 +CPU=  7.946
  02:44:01 STEP  333 H=  83:15 +CPU=  7.865
  02:44:09 STEP  334 H=  83:30 +CPU=  7.728
  02:44:17 STEP  335 H=  83:45 +CPU=  7.832
  02:44:28 STEP  336 H=  84:00 +CPU= 11.171
  02:44:36 STEP  337 H=  84:15 +CPU=  7.575
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000336
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000336
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000336
  02:44:44 STEP  338 H=  84:30 +CPU=  7.668
  02:44:51 STEP  339 H=  84:45 +CPU=  7.815
  02:45:00 STEP  340 H=  85:00 +CPU=  8.145
  02:45:07 STEP  341 H=  85:15 +CPU=  7.602
  02:45:15 STEP  342 H=  85:30 +CPU=  7.860
  02:45:23 STEP  343 H=  85:45 +CPU=  7.767
  02:45:31 STEP  344 H=  86:00 +CPU=  7.909
  02:45:38 STEP  345 H=  86:15 +CPU=  7.671
  02:45:46 STEP  346 H=  86:30 +CPU=  7.833
  02:45:54 STEP  347 H=  86:45 +CPU=  7.842
  02:46:04 STEP  348 H=  87:00 +CPU= 10.359
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000348
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000348
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000348
  02:46:12 STEP  349 H=  87:15 +CPU=  7.356
  02:46:19 STEP  350 H=  87:30 +CPU=  7.595
  02:46:27 STEP  351 H=  87:45 +CPU=  7.804
  02:46:35 STEP  352 H=  88:00 +CPU=  8.189
  02:46:43 STEP  353 H=  88:15 +CPU=  7.752
  02:46:51 STEP  354 H=  88:30 +CPU=  7.823
  02:46:59 STEP  355 H=  88:45 +CPU=  7.707
  02:47:07 STEP  356 H=  89:00 +CPU=  8.097
  02:47:15 STEP  357 H=  89:15 +CPU=  7.803
  02:47:22 STEP  358 H=  89:30 +CPU=  7.641
  02:47:30 STEP  359 H=  89:45 +CPU=  7.900
  02:47:40 STEP  360 H=  90:00 +CPU=  9.743
  02:47:47 STEP  361 H=  90:15 +CPU=  7.416
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000360
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000360
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000360
  02:47:55 STEP  362 H=  90:30 +CPU=  7.285
  02:48:02 STEP  363 H=  90:45 +CPU=  7.371
  02:48:10 STEP  364 H=  91:00 +CPU=  7.898
  02:48:18 STEP  365 H=  91:15 +CPU=  7.692
  02:48:25 STEP  366 H=  91:30 +CPU=  7.722
  02:48:33 STEP  367 H=  91:45 +CPU=  7.608
  02:48:41 STEP  368 H=  92:00 +CPU=  7.868
  02:48:49 STEP  369 H=  92:15 +CPU=  7.623
  02:48:56 STEP  370 H=  92:30 +CPU=  7.327
  02:49:03 STEP  371 H=  92:45 +CPU=  7.477
  02:49:13 STEP  372 H=  93:00 +CPU=  9.984
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000372
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000372
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000372
  02:49:21 STEP  373 H=  93:15 +CPU=  7.667
  02:49:29 STEP  374 H=  93:30 +CPU=  7.885
  02:49:37 STEP  375 H=  93:45 +CPU=  7.851
  02:49:45 STEP  376 H=  94:00 +CPU=  8.328
  02:49:53 STEP  377 H=  94:15 +CPU=  8.055
  02:50:01 STEP  378 H=  94:30 +CPU=  7.869
  02:50:09 STEP  379 H=  94:45 +CPU=  7.546
  02:50:17 STEP  380 H=  95:00 +CPU=  7.913
  02:50:24 STEP  381 H=  95:15 +CPU=  7.737
  02:50:32 STEP  382 H=  95:30 +CPU=  7.778
  02:50:40 STEP  383 H=  95:45 +CPU=  7.681
  02:50:50 STEP  384 H=  96:00 +CPU= 10.499
  02:50:58 STEP  385 H=  96:15 +CPU=  7.422
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000384
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000384
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000384
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000288
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000288
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000288
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000300
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000300
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000300
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000312
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000312
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000312
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000324
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000324
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000324
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000336
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000336
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000336
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000348
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000348
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000348
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000360
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000360
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000360
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000372
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000372
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000372
Zipping up the intermediate file: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_a36f_2016092300_20_1018_12292446_0_r424306766_3.zip
Uploading the intermediate file: upload_file_3.zip
  02:51:06 STEP  386 H=  96:30 +CPU=  7.852
  02:51:13 STEP  387 H=  96:45 +CPU=  7.716
  02:51:21 STEP  388 H=  97:00 +CPU=  7.889
  02:51:29 STEP  389 H=  97:15 +CPU=  7.672
  02:51:37 STEP  390 H=  97:30 +CPU=  7.772
  02:51:45 STEP  391 H=  97:45 +CPU=  8.164
  02:51:53 STEP  392 H=  98:00 +CPU=  7.754
  02:52:00 STEP  393 H=  98:15 +CPU=  7.760
  02:52:08 STEP  394 H=  98:30 +CPU=  7.885
  02:52:16 STEP  395 H=  98:45 +CPU=  7.778
  02:52:26 STEP  396 H=  99:00 +CPU= 10.249
  02:52:34 STEP  397 H=  99:15 +CPU=  7.709
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000396
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000396
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000396
  02:52:42 STEP  398 H=  99:30 +CPU=  8.005
  02:52:50 STEP  399 H=  99:45 +CPU=  7.648
  02:52:58 STEP  400 H= 100:00 +CPU=  7.919
  02:53:05 STEP  401 H= 100:15 +CPU=  7.748
  02:53:13 STEP  402 H= 100:30 +CPU=  7.678
  02:53:21 STEP  403 H= 100:45 +CPU=  7.864
  02:53:29 STEP  404 H= 101:00 +CPU=  7.800
  02:53:36 STEP  405 H= 101:15 +CPU=  7.402
  02:53:44 STEP  406 H= 101:30 +CPU=  7.428
  02:53:51 STEP  407 H= 101:45 +CPU=  7.368
  02:54:01 STEP  408 H= 102:00 +CPU=  9.994
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000408
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000408
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000408
  02:54:09 STEP  409 H= 102:15 +CPU=  7.767
  02:54:17 STEP  410 H= 102:30 +CPU=  7.709
  02:54:25 STEP  411 H= 102:45 +CPU=  7.905
  02:54:33 STEP  412 H= 103:00 +CPU=  8.224
  02:54:41 STEP  413 H= 103:15 +CPU=  7.970
  02:54:48 STEP  414 H= 103:30 +CPU=  7.736
  02:54:56 STEP  415 H= 103:45 +CPU=  7.535
  02:55:04 STEP  416 H= 104:00 +CPU=  8.192
  02:55:12 STEP  417 H= 104:15 +CPU=  7.874
  02:55:20 STEP  418 H= 104:30 +CPU=  7.656
  02:55:28 STEP  419 H= 104:45 +CPU=  7.833
  02:55:38 STEP  420 H= 105:00 +CPU= 10.003
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000420
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000420
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000420
  02:55:45 STEP  421 H= 105:15 +CPU=  7.724
  02:55:53 STEP  422 H= 105:30 +CPU=  7.651
  02:56:01 STEP  423 H= 105:45 +CPU=  7.559
  02:56:09 STEP  424 H= 106:00 +CPU=  7.980
  02:56:16 STEP  425 H= 106:15 +CPU=  7.866
  02:56:24 STEP  426 H= 106:30 +CPU=  7.795
  02:56:32 STEP  427 H= 106:45 +CPU=  7.788
  02:56:40 STEP  428 H= 107:00 +CPU=  7.730
  02:56:48 STEP  429 H= 107:15 +CPU=  8.191
  02:56:56 STEP  430 H= 107:30 +CPU=  7.607
  02:57:03 STEP  431 H= 107:45 +CPU=  7.373
  02:57:14 STEP  432 H= 108:00 +CPU= 10.864
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000432
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000432
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000432
  02:57:22 STEP  433 H= 108:15 +CPU=  7.892
  02:57:30 STEP  434 H= 108:30 +CPU=  7.871
  02:57:37 STEP  435 H= 108:45 +CPU=  7.544
  02:57:45 STEP  436 H= 109:00 +CPU=  8.017
  02:57:53 STEP  437 H= 109:15 +CPU=  7.712
  02:58:01 STEP  438 H= 109:30 +CPU=  7.567
  02:58:09 STEP  439 H= 109:45 +CPU=  7.923
  02:58:17 STEP  440 H= 110:00 +CPU=  8.067
  02:58:25 STEP  441 H= 110:15 +CPU=  7.920
  02:58:32 STEP  442 H= 110:30 +CPU=  7.709
  02:58:40 STEP  443 H= 110:45 +CPU=  7.772
  02:58:50 STEP  444 H= 111:00 +CPU= 10.021
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000444
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000444
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000444
  02:58:58 STEP  445 H= 111:15 +CPU=  7.580
  02:59:06 STEP  446 H= 111:30 +CPU=  7.852
  02:59:13 STEP  447 H= 111:45 +CPU=  7.724
  02:59:21 STEP  448 H= 112:00 +CPU=  7.981
  02:59:29 STEP  449 H= 112:15 +CPU=  7.726
  02:59:37 STEP  450 H= 112:30 +CPU=  7.793
  02:59:44 STEP  451 H= 112:45 +CPU=  7.456
  02:59:53 STEP  452 H= 113:00 +CPU=  8.216
  03:00:00 STEP  453 H= 113:15 +CPU=  7.914
  03:00:08 STEP  454 H= 113:30 +CPU=  7.379
  03:00:16 STEP  455 H= 113:45 +CPU=  7.935
  03:00:26 STEP  456 H= 114:00 +CPU= 10.100
  03:00:33 STEP  457 H= 114:15 +CPU=  7.509
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000456
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000456
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000456
  03:00:41 STEP  458 H= 114:30 +CPU=  7.656
  03:00:49 STEP  459 H= 114:45 +CPU=  7.666
  03:00:57 STEP  460 H= 115:00 +CPU=  7.942
  03:01:04 STEP  461 H= 115:15 +CPU=  7.354
  03:01:11 STEP  462 H= 115:30 +CPU=  7.399
  03:01:19 STEP  463 H= 115:45 +CPU=  7.368
  03:01:27 STEP  464 H= 116:00 +CPU=  8.050
  03:01:35 STEP  465 H= 116:15 +CPU=  7.745
  03:01:42 STEP  466 H= 116:30 +CPU=  7.679
  03:01:50 STEP  467 H= 116:45 +CPU=  7.528
  03:02:00 STEP  468 H= 117:00 +CPU=  9.684
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000468
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000468
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000468
  03:02:07 STEP  469 H= 117:15 +CPU=  7.769
  03:02:15 STEP  470 H= 117:30 +CPU=  7.616
  03:02:22 STEP  471 H= 117:45 +CPU=  7.226
  03:02:30 STEP  472 H= 118:00 +CPU=  7.624
  03:02:38 STEP  473 H= 118:15 +CPU=  7.601
  03:02:45 STEP  474 H= 118:30 +CPU=  7.660
  03:02:53 STEP  475 H= 118:45 +CPU=  7.842
  03:03:01 STEP  476 H= 119:00 +CPU=  7.934
  03:03:09 STEP  477 H= 119:15 +CPU=  7.853
  03:03:17 STEP  478 H= 119:30 +CPU=  7.746
  03:03:25 STEP  479 H= 119:45 +CPU=  7.994
  03:03:36 STEP  480 H= 120:00 +CPU= 11.557
  03:03:44 STEP  481 H= 120:15 +CPU=  7.881
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000480
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000480
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000480
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000384
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000384
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000384
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000396
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000396
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000396
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000408
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000408
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000408
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000420
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000420
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000420
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000432
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000432
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000432
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000444
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000444
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000444
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000456
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000456
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000456
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMGGh7zg+000468
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMSHh7zg+000468
Adding to the zip: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_12292446/ICMUAh7zg+000468
Zipping up the intermediate file: /mnt/spin/nobak/boinc/boinc-client-cpdn/projects/climateprediction.net/oifs_43r3_bl_a36f_2016092300_20_1018_12292446_0_r424306766_4.zip
Uploading the intermediate file: upload_file_4.zip
  03:03:52 STEP  482 H= 120:30 +CPU=  8.189
  03:04:00 STEP  483 H= 120:45 +CPU=  7.865
  03:04:08 STEP  484 H= 121:00 +CPU=  7.947
  03:04:15 STEP  485 H= 121:15 +CPU=  7.237
  03:04:23 STEP  486 H= 121:30 +CPU=  7.587
  03:04:31 STEP  487 H= 121:45 +CPU=  7.570
  03:04:38 STEP  488 H= 122:00 +CPU=  7.785
  03:04:46 STEP  489 H= 122:15 +CPU=  7.449
  03:04:54 STEP  490 H= 122:30 +CPU=  7.832
  03:05:02 STEP  491 H= 122:45 +CPU=  8.108
  03:05:12 STEP  492 H= 123:00 +CPU= 10.080
  03:05:19 STEP  493 H= 123:15 +CPU=  7.556
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000492
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000492
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000492
  03:05:27 STEP  494 H= 123:30 +CPU=  7.750
  03:05:35 STEP  495 H= 123:45 +CPU=  8.059
  03:05:43 STEP  496 H= 124:00 +CPU=  7.856
  03:05:51 STEP  497 H= 124:15 +CPU=  8.163
  03:05:59 STEP  498 H= 124:30 +CPU=  8.022
  03:06:07 STEP  499 H= 124:45 +CPU=  7.297
  03:06:14 STEP  500 H= 125:00 +CPU=  7.784
  03:06:22 STEP  501 H= 125:15 +CPU=  7.671
  03:06:30 STEP  502 H= 125:30 +CPU=  7.584
  03:06:37 STEP  503 H= 125:45 +CPU=  7.377
  03:06:47 STEP  504 H= 126:00 +CPU= 10.106
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000504
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000504
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000504
  03:06:55 STEP  505 H= 126:15 +CPU=  7.547
  03:07:02 STEP  506 H= 126:30 +CPU=  7.587
  03:07:10 STEP  507 H= 126:45 +CPU=  7.711
  03:07:18 STEP  508 H= 127:00 +CPU=  8.151
  03:07:26 STEP  509 H= 127:15 +CPU=  7.782
  03:07:34 STEP  510 H= 127:30 +CPU=  7.555
  03:07:41 STEP  511 H= 127:45 +CPU=  7.738
  03:07:49 STEP  512 H= 128:00 +CPU=  7.967
  03:07:57 STEP  513 H= 128:15 +CPU=  7.466
  03:08:05 STEP  514 H= 128:30 +CPU=  7.703
  03:08:13 STEP  515 H= 128:45 +CPU=  8.017
  03:08:23 STEP  516 H= 129:00 +CPU= 10.181
  03:08:30 STEP  517 H= 129:15 +CPU=  7.508
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMGGh7zg+000516
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMSHh7zg+000516
Moving to projects directory: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/ICMUAh7zg+000516
  03:08:38 STEP  518 H= 129:30 +CPU=  7.616
  03:08:46 STEP  519 H= 129:45 +CPU=  7.574
  03:08:53 STEP  520 H= 130:00 +CPU=  7.806
  03:09:01 STEP  521 H= 130:15 +CPU=  7.318
  03:09:08 STEP  522 H= 130:30 +CPU=  7.428
  03:09:16 STEP  523 H= 130:45 +CPU=  7.443
  03:09:23 STEP  524 H= 131:00 +CPU=  7.662
  03:09:31 STEP  525 H= 131:15 +CPU=  7.665
  03:09:39 STEP  526 H= 131:30 +CPU=  7.792
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5555MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5555 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.154] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030944:1718608184:4117.155] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/oifs_43r3_model.exe' : sigcontextptr=0x7fffe5da6950
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.155] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/oifs_43r3_model.exe' : sigcontextptr=0x7fffe5da6860
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.162] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:ilia:1:1:135972:135972] [20240617:030945:1718608185:4118.162] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.136933618986438E-04 0.000000000000000E+00 0.580068820620110E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.537853885017246E-05 0.000000000000000E+00 0.154942072861849E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.100073260289462E-05 0.000000000000000E+00 0.967943346990827E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.143649427084937E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =   516 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   516 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000516 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+000516
 MPI-TASK:   1 -   1243045604 BYTES IN      6 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0029( 6.9%)
 NSTEP =   516 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+000516 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+000516
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0024 SECONDS294749.7 Mbytes/s, TOTAL TIME=    0.0907( 2.6%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000516 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+000516
 MPI-TASK:   1 -  19888729664 BYTES IN    125 RECORDS TRANSFERRED IN    0.0019 SECONDS******** Mbytes/s, TOTAL TIME=    0.0643( 3.0%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000516 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000516
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0000 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0015( 0.0%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+000516 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+000516
 MPI-TASK:   1 -   1234465875 BYTES IN      4 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0022( 4.5%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =   516 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =   516 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+000516 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+000516
 MPI-TASK:   1 -   1243045604 BYTES IN      7 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0031( 3.2%)
 03:08:23 STEP  516 H= 129:00 +CPU= 10.181
 NSTEP =   517 STEPO   0AAA00AAA
 03:08:30 STEP  517 H= 129:15 +CPU=  7.508
 NSTEP =   518 STEPO   0AAA00AAA
 03:08:38 STEP  518 H= 129:30 +CPU=  7.616
 NSTEP =   519 STEPO   0AAA00AAA
 03:08:46 STEP  519 H= 129:45 +CPU=  7.574
ISTEP=     520 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     520 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   520 STEPO   0AAA00AAA
 03:08:53 STEP  520 H= 130:00 +CPU=  7.806
 NSTEP =   521 STEPO   0AAA00AAA
 03:09:01 STEP  521 H= 130:15 +CPU=  7.318
 NSTEP =   522 STEPO   0AAA00AAA
 03:09:08 STEP  522 H= 130:30 +CPU=  7.428
 NSTEP =   523 STEPO   0AAA00AAA
 03:09:16 STEP  523 H= 130:45 +CPU=  7.443
 NSTEP =   524 STEPO   0AAA00AAA
 03:09:23 STEP  524 H= 131:00 +CPU=  7.662
ISTEP=     525 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=     525 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =   525 STEPO   0AAA00AAA
 03:09:31 STEP  525 H= 131:15 +CPU=  7.665
 NSTEP =   526 STEPO   0AAA00AAA
 03:09:39 STEP  526 H= 131:30 +CPU=  7.792
 NSTEP =   527 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 03:08:53 0AAA00AAA STEPO      521     7.806    7.806    7.815     67:40     67:46 0.33296577162986E-04       2GB       0MB
 03:09:01 0AAA00AAA STEPO      522     7.318    7.318    7.329     67:47     67:53 0.33296577162986E-04       2GB       0MB
 03:09:08 0AAA00AAA STEPO      523     7.428    7.428    7.440     67:55     68:01 0.33296577162986E-04       2GB       0MB
 03:09:16 0AAA00AAA STEPO      524     7.443    7.443    7.457     68:02     68:08 0.33296577162986E-04       2GB       0MB
 03:09:23 0AAA00AAA STEPO      525     7.662    7.662    7.670     68:10     68:16 0.33296577162986E-04       2GB       0MB
 03:09:31 0AAA00AAA STEPO      526     7.665    7.665    7.675     68:17     68:24 0.33296577162986E-04       2GB       0MB
 03:09:39 0AAA00AAA STEPO      527     7.793    7.793    7.809     68:25     68:31 0.33296577162986E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "     480",
 CTIME   = "00050000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         468, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /mnt/spin/nobak/boinc/boinc-client-cpdn/slots/0/progress_file_12292446.xml

<running_values>
  <last_cpu_time>4107.960000</last_cpu_time>
  <upload_file_number>5</upload_file_number>
  <last_iter>527</last_iter>
  <last_upload>432000</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net