climateprediction.net home page
Task 22435682

Task 22435682

Name oifs_43r3_bl_a1r7_2016092300_20_1017_12284102_0
Workunit 12284102
Created 7 Jun 2024, 14:08:12 UTC
Sent 7 Jun 2024, 14:16:51 UTC
Report deadline 6 Aug 2024, 14:16:51 UTC
Received 8 Jun 2024, 5:13:51 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538899
Run time 14 hours 43 min 52 sec
CPU time 14 hours 37 min 32 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 3.88 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.23 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,581.73 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
:15 +CPU= 47.081
  17:55:39 STEP  690 H= 172:30 +CPU= 51.943
  17:56:32 STEP  691 H= 172:45 +CPU= 52.807
  17:57:25 STEP  692 H= 173:00 +CPU= 53.067
  17:58:17 STEP  693 H= 173:15 +CPU= 51.835
  17:59:09 STEP  694 H= 173:30 +CPU= 51.680
  18:00:00 STEP  695 H= 173:45 +CPU= 50.760
  18:01:04 STEP  696 H= 174:00 +CPU= 63.998
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000696
  18:01:56 STEP  697 H= 174:15 +CPU= 52.042
  18:02:49 STEP  698 H= 174:30 +CPU= 52.267
  18:03:40 STEP  699 H= 174:45 +CPU= 50.580
  18:04:33 STEP  700 H= 175:00 +CPU= 52.635
  18:05:25 STEP  701 H= 175:15 +CPU= 51.332
  18:06:18 STEP  702 H= 175:30 +CPU= 51.493
  18:07:08 STEP  703 H= 175:45 +CPU= 50.776
  18:08:01 STEP  704 H= 176:00 +CPU= 52.300
  18:08:52 STEP  705 H= 176:15 +CPU= 50.918
  18:09:45 STEP  706 H= 176:30 +CPU= 51.629
  18:10:37 STEP  707 H= 176:45 +CPU= 51.663
  18:11:42 STEP  708 H= 177:00 +CPU= 64.776
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000708
  18:12:34 STEP  709 H= 177:15 +CPU= 51.655
  18:13:25 STEP  710 H= 177:30 +CPU= 51.174
  18:14:16 STEP  711 H= 177:45 +CPU= 49.970
  18:15:09 STEP  712 H= 178:00 +CPU= 52.834
  18:16:01 STEP  713 H= 178:15 +CPU= 50.905
  18:16:53 STEP  714 H= 178:30 +CPU= 51.763
  18:17:44 STEP  715 H= 178:45 +CPU= 51.195
  18:18:35 STEP  716 H= 179:00 +CPU= 50.632
  18:19:26 STEP  717 H= 179:15 +CPU= 50.351
  18:20:20 STEP  718 H= 179:30 +CPU= 50.797
  18:21:10 STEP  719 H= 179:45 +CPU= 49.752
  18:22:19 STEP  720 H= 180:00 +CPU= 68.957
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000720
  18:23:11 STEP  721 H= 180:15 +CPU= 51.510
  18:24:02 STEP  722 H= 180:30 +CPU= 51.510
  18:24:50 STEP  723 H= 180:45 +CPU= 47.802
  18:25:44 STEP  724 H= 181:00 +CPU= 53.141
  18:26:36 STEP  725 H= 181:15 +CPU= 52.660
  18:27:28 STEP  726 H= 181:30 +CPU= 52.085
  18:28:20 STEP  727 H= 181:45 +CPU= 51.445
  18:29:12 STEP  728 H= 182:00 +CPU= 52.313
  18:30:06 STEP  729 H= 182:15 +CPU= 51.895
  18:30:58 STEP  730 H= 182:30 +CPU= 51.932
  18:31:50 STEP  731 H= 182:45 +CPU= 51.670
  18:32:54 STEP  732 H= 183:00 +CPU= 63.766
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000732
  18:33:46 STEP  733 H= 183:15 +CPU= 51.279
  18:34:38 STEP  734 H= 183:30 +CPU= 51.571
  18:35:31 STEP  735 H= 183:45 +CPU= 53.004
  18:36:25 STEP  736 H= 184:00 +CPU= 53.411
  18:37:18 STEP  737 H= 184:15 +CPU= 52.428
  18:38:10 STEP  738 H= 184:30 +CPU= 52.206
  18:39:02 STEP  739 H= 184:45 +CPU= 51.925
  18:39:56 STEP  740 H= 185:00 +CPU= 53.238
  18:40:46 STEP  741 H= 185:15 +CPU= 50.529
  18:41:38 STEP  742 H= 185:30 +CPU= 50.924
  18:42:30 STEP  743 H= 185:45 +CPU= 51.957
  18:43:35 STEP  744 H= 186:00 +CPU= 63.622
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000744
  18:44:24 STEP  745 H= 186:15 +CPU= 49.620
  18:45:17 STEP  746 H= 186:30 +CPU= 51.287
  18:46:10 STEP  747 H= 186:45 +CPU= 52.053
  18:47:03 STEP  748 H= 187:00 +CPU= 52.848
  18:47:55 STEP  749 H= 187:15 +CPU= 51.699
  18:48:47 STEP  750 H= 187:30 +CPU= 52.165
  18:49:39 STEP  751 H= 187:45 +CPU= 51.938
  18:50:32 STEP  752 H= 188:00 +CPU= 52.529
  18:51:22 STEP  753 H= 188:15 +CPU= 49.732
  18:52:13 STEP  754 H= 188:30 +CPU= 50.196
  18:53:03 STEP  755 H= 188:45 +CPU= 50.106
  18:54:03 STEP  756 H= 189:00 +CPU= 60.307
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000756
  18:54:53 STEP  757 H= 189:15 +CPU= 49.658
  18:55:43 STEP  758 H= 189:30 +CPU= 49.778
  18:56:30 STEP  759 H= 189:45 +CPU= 46.573
  18:57:20 STEP  760 H= 190:00 +CPU= 50.350
  18:58:11 STEP  761 H= 190:15 +CPU= 50.592
  18:59:01 STEP  762 H= 190:30 +CPU= 50.314
  18:59:51 STEP  763 H= 190:45 +CPU= 49.771
  19:00:41 STEP  764 H= 191:00 +CPU= 49.307
  19:01:31 STEP  765 H= 191:15 +CPU= 50.326
  19:02:22 STEP  766 H= 191:30 +CPU= 50.234
  19:03:13 STEP  767 H= 191:45 +CPU= 50.542
  19:04:18 STEP  768 H= 192:00 +CPU= 64.998
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1r7_2016092300_20_1017_12284102_0_r1922988982_7.zip
Uploading the intermediate file: upload_file_7.zip
  19:05:09 STEP  769 H= 192:15 +CPU= 49.282
  19:05:55 STEP  770 H= 192:30 +CPU= 46.431
  19:06:45 STEP  771 H= 192:45 +CPU= 49.213
  19:07:34 STEP  772 H= 193:00 +CPU= 49.449
  19:08:25 STEP  773 H= 193:15 +CPU= 50.236
  19:09:12 STEP  774 H= 193:30 +CPU= 46.935
  19:10:02 STEP  775 H= 193:45 +CPU= 50.349
  19:10:54 STEP  776 H= 194:00 +CPU= 50.717
  19:11:42 STEP  777 H= 194:15 +CPU= 47.633
  19:12:32 STEP  778 H= 194:30 +CPU= 49.979
  19:13:22 STEP  779 H= 194:45 +CPU= 49.793
  19:14:23 STEP  780 H= 195:00 +CPU= 61.321
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000780
  19:15:10 STEP  781 H= 195:15 +CPU= 46.559
  19:15:59 STEP  782 H= 195:30 +CPU= 49.676
  19:16:46 STEP  783 H= 195:45 +CPU= 46.305
  19:17:36 STEP  784 H= 196:00 +CPU= 50.362
  19:18:26 STEP  785 H= 196:15 +CPU= 48.273
  19:19:16 STEP  786 H= 196:30 +CPU= 50.191
  19:20:03 STEP  787 H= 196:45 +CPU= 47.365
  19:20:55 STEP  788 H= 197:00 +CPU= 50.459
  19:21:45 STEP  789 H= 197:15 +CPU= 49.278
  19:22:34 STEP  790 H= 197:30 +CPU= 48.976
  19:23:24 STEP  791 H= 197:45 +CPU= 49.928
  19:24:27 STEP  792 H= 198:00 +CPU= 62.561
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000792
  19:25:15 STEP  793 H= 198:15 +CPU= 48.522
  19:26:06 STEP  794 H= 198:30 +CPU= 50.941
  19:26:56 STEP  795 H= 198:45 +CPU= 48.904
  19:27:48 STEP  796 H= 199:00 +CPU= 51.946
  19:28:38 STEP  797 H= 199:15 +CPU= 49.739
  19:29:26 STEP  798 H= 199:30 +CPU= 47.502
  19:30:13 STEP  799 H= 199:45 +CPU= 47.289
  19:31:02 STEP  800 H= 200:00 +CPU= 48.658
  19:31:48 STEP  801 H= 200:15 +CPU= 46.099
  19:32:40 STEP  802 H= 200:30 +CPU= 50.805
  19:33:28 STEP  803 H= 200:45 +CPU= 48.493
  19:34:30 STEP  804 H= 201:00 +CPU= 61.156
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000804
  19:35:20 STEP  805 H= 201:15 +CPU= 50.348
  19:36:09 STEP  806 H= 201:30 +CPU= 48.808
  19:37:00 STEP  807 H= 201:45 +CPU= 49.532
  19:37:53 STEP  808 H= 202:00 +CPU= 52.103
  19:38:44 STEP  809 H= 202:15 +CPU= 51.013
  19:39:36 STEP  810 H= 202:30 +CPU= 51.167
  19:40:25 STEP  811 H= 202:45 +CPU= 49.170
  19:41:17 STEP  812 H= 203:00 +CPU= 51.589
  19:42:07 STEP  813 H= 203:15 +CPU= 50.351
  19:42:57 STEP  814 H= 203:30 +CPU= 49.804
  19:43:47 STEP  815 H= 203:45 +CPU= 50.032
  19:44:54 STEP  816 H= 204:00 +CPU= 66.800
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000816
  19:45:44 STEP  817 H= 204:15 +CPU= 49.325
  19:46:34 STEP  818 H= 204:30 +CPU= 50.155
  19:47:25 STEP  819 H= 204:45 +CPU= 50.714
  19:48:16 STEP  820 H= 205:00 +CPU= 50.750
  19:49:06 STEP  821 H= 205:15 +CPU= 49.552
  19:49:56 STEP  822 H= 205:30 +CPU= 49.870
  19:50:45 STEP  823 H= 205:45 +CPU= 49.253
  19:51:37 STEP  824 H= 206:00 +CPU= 50.978
  19:52:27 STEP  825 H= 206:15 +CPU= 50.679
  19:53:19 STEP  826 H= 206:30 +CPU= 51.076
  19:54:08 STEP  827 H= 206:45 +CPU= 49.071
  19:55:08 STEP  828 H= 207:00 +CPU= 60.368
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000828
  19:55:59 STEP  829 H= 207:15 +CPU= 50.264
  19:56:50 STEP  830 H= 207:30 +CPU= 50.097
  19:57:40 STEP  831 H= 207:45 +CPU= 49.912
  19:58:30 STEP  832 H= 208:00 +CPU= 50.241
  19:59:20 STEP  833 H= 208:15 +CPU= 49.368
  20:00:10 STEP  834 H= 208:30 +CPU= 49.358
  20:00:58 STEP  835 H= 208:45 +CPU= 47.505
  20:01:48 STEP  836 H= 209:00 +CPU= 48.776
  20:02:36 STEP  837 H= 209:15 +CPU= 48.236
  20:03:26 STEP  838 H= 209:30 +CPU= 49.838
  20:04:15 STEP  839 H= 209:45 +CPU= 48.533
  20:05:17 STEP  840 H= 210:00 +CPU= 62.244
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000840
  20:06:07 STEP  841 H= 210:15 +CPU= 49.261
  20:06:56 STEP  842 H= 210:30 +CPU= 49.135
  20:07:45 STEP  843 H= 210:45 +CPU= 48.639
  20:08:34 STEP  844 H= 211:00 +CPU= 48.426
  20:09:24 STEP  845 H= 211:15 +CPU= 50.279
  20:10:12 STEP  846 H= 211:30 +CPU= 47.594
  20:11:02 STEP  847 H= 211:45 +CPU= 49.878
  20:11:51 STEP  848 H= 212:00 +CPU= 49.167
  20:12:40 STEP  849 H= 212:15 +CPU= 48.619
  20:13:30 STEP  850 H= 212:30 +CPU= 50.210
  20:14:19 STEP  851 H= 212:45 +CPU= 48.733
  20:15:19 STEP  852 H= 213:00 +CPU= 60.093
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000852
  20:16:06 STEP  853 H= 213:15 +CPU= 46.873
  20:16:55 STEP  854 H= 213:30 +CPU= 48.873
  20:17:45 STEP  855 H= 213:45 +CPU= 49.828
  20:18:34 STEP  856 H= 214:00 +CPU= 48.438
  20:19:24 STEP  857 H= 214:15 +CPU= 49.881
  20:20:12 STEP  858 H= 214:30 +CPU= 48.444
  20:21:03 STEP  859 H= 214:45 +CPU= 49.857
  20:21:54 STEP  860 H= 215:00 +CPU= 50.916
  20:22:43 STEP  861 H= 215:15 +CPU= 49.176
  20:23:35 STEP  862 H= 215:30 +CPU= 50.084
  20:24:23 STEP  863 H= 215:45 +CPU= 47.821
  20:25:30 STEP  864 H= 216:00 +CPU= 66.327
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1r7_2016092300_20_1017_12284102_0_r1922988982_8.zip
Uploading the intermediate file: upload_file_8.zip
  20:26:21 STEP  865 H= 216:15 +CPU= 49.970
  20:27:11 STEP  866 H= 216:30 +CPU= 49.926
  20:27:59 STEP  867 H= 216:45 +CPU= 47.994
  20:28:49 STEP  868 H= 217:00 +CPU= 50.288
  20:29:40 STEP  869 H= 217:15 +CPU= 50.453
  20:30:28 STEP  870 H= 217:30 +CPU= 48.220
  20:31:19 STEP  871 H= 217:45 +CPU= 50.360
  20:32:06 STEP  872 H= 218:00 +CPU= 46.891
  20:32:55 STEP  873 H= 218:15 +CPU= 49.152
  20:33:45 STEP  874 H= 218:30 +CPU= 49.639
  20:34:35 STEP  875 H= 218:45 +CPU= 49.097
  20:35:33 STEP  876 H= 219:00 +CPU= 57.584
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000876
  20:36:23 STEP  877 H= 219:15 +CPU= 49.709
  20:37:08 STEP  878 H= 219:30 +CPU= 44.927
  20:37:56 STEP  879 H= 219:45 +CPU= 47.347
  20:38:46 STEP  880 H= 220:00 +CPU= 50.239
  20:39:36 STEP  881 H= 220:15 +CPU= 49.688
  20:40:25 STEP  882 H= 220:30 +CPU= 48.842
  20:41:14 STEP  883 H= 220:45 +CPU= 48.969
  20:42:04 STEP  884 H= 221:00 +CPU= 49.536
  20:42:52 STEP  885 H= 221:15 +CPU= 47.703
  20:43:41 STEP  886 H= 221:30 +CPU= 49.344
  20:44:31 STEP  887 H= 221:45 +CPU= 49.236
  20:45:29 STEP  888 H= 222:00 +CPU= 58.007
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000888
  20:46:16 STEP  889 H= 222:15 +CPU= 47.382
  20:47:04 STEP  890 H= 222:30 +CPU= 46.863
  20:47:51 STEP  891 H= 222:45 +CPU= 47.654
  20:48:40 STEP  892 H= 223:00 +CPU= 47.734
  20:49:29 STEP  893 H= 223:15 +CPU= 49.199
  20:50:15 STEP  894 H= 223:30 +CPU= 45.501
  20:51:02 STEP  895 H= 223:45 +CPU= 46.227
  20:51:49 STEP  896 H= 224:00 +CPU= 46.815
  20:52:35 STEP  897 H= 224:15 +CPU= 45.738
  20:53:21 STEP  898 H= 224:30 +CPU= 45.796
  20:54:06 STEP  899 H= 224:45 +CPU= 44.562
  20:55:03 STEP  900 H= 225:00 +CPU= 57.091
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000900
  20:55:50 STEP  901 H= 225:15 +CPU= 46.543
  20:56:37 STEP  902 H= 225:30 +CPU= 47.352
  20:57:25 STEP  903 H= 225:45 +CPU= 47.908
  20:58:14 STEP  904 H= 226:00 +CPU= 48.513
  20:58:58 STEP  905 H= 226:15 +CPU= 44.633
  20:59:47 STEP  906 H= 226:30 +CPU= 48.225
  21:00:33 STEP  907 H= 226:45 +CPU= 45.742
  21:01:20 STEP  908 H= 227:00 +CPU= 46.947
  21:02:07 STEP  909 H= 227:15 +CPU= 46.484
  21:02:55 STEP  910 H= 227:30 +CPU= 48.115
  21:03:45 STEP  911 H= 227:45 +CPU= 50.185
  21:04:54 STEP  912 H= 228:00 +CPU= 68.298
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000912
  21:05:44 STEP  913 H= 228:15 +CPU= 49.892
  21:06:36 STEP  914 H= 228:30 +CPU= 51.432
  21:07:28 STEP  915 H= 228:45 +CPU= 51.938
  21:08:22 STEP  916 H= 229:00 +CPU= 53.395
  21:09:15 STEP  917 H= 229:15 +CPU= 52.981
  21:10:10 STEP  918 H= 229:30 +CPU= 54.561
  21:11:04 STEP  919 H= 229:45 +CPU= 54.110
  21:11:59 STEP  920 H= 230:00 +CPU= 54.945
  21:12:53 STEP  921 H= 230:15 +CPU= 53.919
  21:13:47 STEP  922 H= 230:30 +CPU= 54.071
  21:14:42 STEP  923 H= 230:45 +CPU= 54.688
  21:15:49 STEP  924 H= 231:00 +CPU= 66.984
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000924
  21:16:43 STEP  925 H= 231:15 +CPU= 53.617
  21:17:38 STEP  926 H= 231:30 +CPU= 53.983
  21:18:32 STEP  927 H= 231:45 +CPU= 54.063
  21:19:27 STEP  928 H= 232:00 +CPU= 54.388
  21:20:22 STEP  929 H= 232:15 +CPU= 53.612
  21:21:16 STEP  930 H= 232:30 +CPU= 53.694
  21:22:11 STEP  931 H= 232:45 +CPU= 54.215
  21:23:07 STEP  932 H= 233:00 +CPU= 55.588
  21:24:02 STEP  933 H= 233:15 +CPU= 54.831
  21:24:56 STEP  934 H= 233:30 +CPU= 54.251
  21:25:50 STEP  935 H= 233:45 +CPU= 54.399
  21:26:59 STEP  936 H= 234:00 +CPU= 68.364
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000936
  21:27:54 STEP  937 H= 234:15 +CPU= 54.861
  21:28:48 STEP  938 H= 234:30 +CPU= 54.006
  21:29:42 STEP  939 H= 234:45 +CPU= 53.921
  21:30:38 STEP  940 H= 235:00 +CPU= 55.733
  21:31:32 STEP  941 H= 235:15 +CPU= 54.045
  21:32:26 STEP  942 H= 235:30 +CPU= 54.120
  21:33:21 STEP  943 H= 235:45 +CPU= 54.052
  21:34:16 STEP  944 H= 236:00 +CPU= 55.021
  21:35:11 STEP  945 H= 236:15 +CPU= 54.645
  21:36:05 STEP  946 H= 236:30 +CPU= 54.207
  21:37:00 STEP  947 H= 236:45 +CPU= 54.030
  21:38:06 STEP  948 H= 237:00 +CPU= 65.730
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000948
  21:39:01 STEP  949 H= 237:15 +CPU= 55.296
  21:39:55 STEP  950 H= 237:30 +CPU= 53.948
  21:40:49 STEP  951 H= 237:45 +CPU= 54.088
  21:41:46 STEP  952 H= 238:00 +CPU= 55.937
  21:42:40 STEP  953 H= 238:15 +CPU= 54.571
  21:43:36 STEP  954 H= 238:30 +CPU= 55.031
  21:44:30 STEP  955 H= 238:45 +CPU= 54.528
  21:45:25 STEP  956 H= 239:00 +CPU= 54.931
  21:46:20 STEP  957 H= 239:15 +CPU= 54.066
  21:47:14 STEP  958 H= 239:30 +CPU= 54.173
  21:48:08 STEP  959 H= 239:45 +CPU= 53.564
  21:49:20 STEP  960 H= 240:00 +CPU= 72.158
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1r7_2016092300_20_1017_12284102_0_r1922988982_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  21:50:15 STEP  961 H= 240:15 +CPU= 54.023
  21:51:09 STEP  962 H= 240:30 +CPU= 53.939
  21:52:03 STEP  963 H= 240:45 +CPU= 52.212
  21:52:58 STEP  964 H= 241:00 +CPU= 54.314
  21:53:53 STEP  965 H= 241:15 +CPU= 54.732
  21:54:47 STEP  966 H= 241:30 +CPU= 54.423
  21:55:42 STEP  967 H= 241:45 +CPU= 54.698
  21:56:36 STEP  968 H= 242:00 +CPU= 54.103
  21:57:31 STEP  969 H= 242:15 +CPU= 53.750
  21:58:25 STEP  970 H= 242:30 +CPU= 53.878
  21:59:18 STEP  971 H= 242:45 +CPU= 52.883
  22:00:24 STEP  972 H= 243:00 +CPU= 65.273
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000972
  22:01:19 STEP  973 H= 243:15 +CPU= 53.560
  22:02:13 STEP  974 H= 243:30 +CPU= 54.037
  22:03:07 STEP  975 H= 243:45 +CPU= 52.969
  22:04:02 STEP  976 H= 244:00 +CPU= 54.643
  22:04:57 STEP  977 H= 244:15 +CPU= 54.659
  22:05:52 STEP  978 H= 244:30 +CPU= 55.003
  22:06:48 STEP  979 H= 244:45 +CPU= 55.052
  22:07:43 STEP  980 H= 245:00 +CPU= 54.956
  22:08:36 STEP  981 H= 245:15 +CPU= 52.694
  22:09:30 STEP  982 H= 245:30 +CPU= 53.687
  22:10:24 STEP  983 H= 245:45 +CPU= 52.452
  22:11:30 STEP  984 H= 246:00 +CPU= 64.372
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000984
  22:12:24 STEP  985 H= 246:15 +CPU= 54.203
  22:13:17 STEP  986 H= 246:30 +CPU= 53.248
  22:14:12 STEP  987 H= 246:45 +CPU= 53.334
  22:15:07 STEP  988 H= 247:00 +CPU= 54.324
  22:16:02 STEP  989 H= 247:15 +CPU= 55.156
  22:16:56 STEP  990 H= 247:30 +CPU= 53.765
  22:17:51 STEP  991 H= 247:45 +CPU= 55.194
  22:18:46 STEP  992 H= 248:00 +CPU= 54.647
  22:19:40 STEP  993 H= 248:15 +CPU= 54.120
  22:20:34 STEP  994 H= 248:30 +CPU= 53.445
  22:21:28 STEP  995 H= 248:45 +CPU= 53.301
  22:22:37 STEP  996 H= 249:00 +CPU= 66.762
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+000996
  22:23:30 STEP  997 H= 249:15 +CPU= 53.462
  22:24:24 STEP  998 H= 249:30 +CPU= 53.950
  22:25:18 STEP  999 H= 249:45 +CPU= 53.411
  22:26:15 STEP 1000 H= 250:00 +CPU= 54.389
  22:27:11 STEP 1001 H= 250:15 +CPU= 55.084
  22:28:04 STEP 1002 H= 250:30 +CPU= 53.042
  22:28:57 STEP 1003 H= 250:45 +CPU= 52.648
  22:29:52 STEP 1004 H= 251:00 +CPU= 54.569
  22:30:46 STEP 1005 H= 251:15 +CPU= 53.735
  22:31:40 STEP 1006 H= 251:30 +CPU= 53.658
  22:32:34 STEP 1007 H= 251:45 +CPU= 53.952
  22:33:48 STEP 1008 H= 252:00 +CPU= 72.749
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001008
  22:34:41 STEP 1009 H= 252:15 +CPU= 52.869
  22:35:36 STEP 1010 H= 252:30 +CPU= 54.225
  22:36:30 STEP 1011 H= 252:45 +CPU= 53.948
  22:37:25 STEP 1012 H= 253:00 +CPU= 54.712
  22:38:20 STEP 1013 H= 253:15 +CPU= 54.667
  22:39:14 STEP 1014 H= 253:30 +CPU= 54.136
  22:40:09 STEP 1015 H= 253:45 +CPU= 54.572
  22:41:04 STEP 1016 H= 254:00 +CPU= 54.633
  22:41:59 STEP 1017 H= 254:15 +CPU= 54.239
  22:42:54 STEP 1018 H= 254:30 +CPU= 54.752
  22:43:48 STEP 1019 H= 254:45 +CPU= 53.748
  22:44:54 STEP 1020 H= 255:00 +CPU= 66.293
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001020
  22:45:48 STEP 1021 H= 255:15 +CPU= 53.956
  22:46:44 STEP 1022 H= 255:30 +CPU= 55.065
  22:47:39 STEP 1023 H= 255:45 +CPU= 54.700
  22:48:34 STEP 1024 H= 256:00 +CPU= 54.477
  22:49:28 STEP 1025 H= 256:15 +CPU= 54.219
  22:50:21 STEP 1026 H= 256:30 +CPU= 53.153
  22:51:16 STEP 1027 H= 256:45 +CPU= 53.996
  22:52:11 STEP 1028 H= 257:00 +CPU= 54.475
  22:53:05 STEP 1029 H= 257:15 +CPU= 54.189
  22:54:00 STEP 1030 H= 257:30 +CPU= 53.952
  22:54:54 STEP 1031 H= 257:45 +CPU= 54.299
  22:56:01 STEP 1032 H= 258:00 +CPU= 66.780
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001032
  22:56:53 STEP 1033 H= 258:15 +CPU= 52.227
  22:57:46 STEP 1034 H= 258:30 +CPU= 52.318
  22:58:38 STEP 1035 H= 258:45 +CPU= 52.093
  22:59:32 STEP 1036 H= 259:00 +CPU= 53.985
  23:00:26 STEP 1037 H= 259:15 +CPU= 53.313
  23:01:19 STEP 1038 H= 259:30 +CPU= 52.790
  23:02:12 STEP 1039 H= 259:45 +CPU= 53.754
  23:03:07 STEP 1040 H= 260:00 +CPU= 53.649
  23:04:01 STEP 1041 H= 260:15 +CPU= 53.415
  23:04:55 STEP 1042 H= 260:30 +CPU= 53.338
  23:05:49 STEP 1043 H= 260:45 +CPU= 53.509
  23:06:56 STEP 1044 H= 261:00 +CPU= 66.330
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001044
  23:07:50 STEP 1045 H= 261:15 +CPU= 53.912
  23:08:45 STEP 1046 H= 261:30 +CPU= 54.792
  23:09:39 STEP 1047 H= 261:45 +CPU= 54.416
  23:10:34 STEP 1048 H= 262:00 +CPU= 54.868
  23:11:28 STEP 1049 H= 262:15 +CPU= 54.036
  23:12:22 STEP 1050 H= 262:30 +CPU= 53.299
  23:13:16 STEP 1051 H= 262:45 +CPU= 52.338
  23:14:10 STEP 1052 H= 263:00 +CPU= 53.787
  23:15:03 STEP 1053 H= 263:15 +CPU= 52.676
  23:15:56 STEP 1054 H= 263:30 +CPU= 52.554
  23:16:49 STEP 1055 H= 263:45 +CPU= 52.793
  23:18:00 STEP 1056 H= 264:00 +CPU= 71.017
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12284102/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1r7_2016092300_20_1017_12284102_0_r1922988982_10.zip
Uploading the intermediate file: upload_file_10.zip
  23:18:54 STEP 1057 H= 264:15 +CPU= 53.915
  23:19:48 STEP 1058 H= 264:30 +CPU= 53.796
  23:20:44 STEP 1059 H= 264:45 +CPU= 54.469
  23:21:39 STEP 1060 H= 265:00 +CPU= 54.382
  23:22:33 STEP 1061 H= 265:15 +CPU= 54.351
  23:23:26 STEP 1062 H= 265:30 +CPU= 52.904
  23:24:20 STEP 1063 H= 265:45 +CPU= 53.015
  23:25:17 STEP 1064 H= 266:00 +CPU= 54.701
  23:26:10 STEP 1065 H= 266:15 +CPU= 53.327
  23:27:03 STEP 1066 H= 266:30 +CPU= 53.296
  23:27:56 STEP 1067 H= 266:45 +CPU= 52.461
  23:29:01 STEP 1068 H= 267:00 +CPU= 65.009
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001068
  23:29:55 STEP 1069 H= 267:15 +CPU= 53.067
  23:30:48 STEP 1070 H= 267:30 +CPU= 53.204
  23:31:42 STEP 1071 H= 267:45 +CPU= 53.158
  23:32:36 STEP 1072 H= 268:00 +CPU= 53.556
  23:33:30 STEP 1073 H= 268:15 +CPU= 52.503
  23:34:22 STEP 1074 H= 268:30 +CPU= 52.408
  23:35:17 STEP 1075 H= 268:45 +CPU= 54.026
  23:36:12 STEP 1076 H= 269:00 +CPU= 54.275
  23:37:06 STEP 1077 H= 269:15 +CPU= 54.129
  23:38:02 STEP 1078 H= 269:30 +CPU= 54.320
  23:38:56 STEP 1079 H= 269:45 +CPU= 53.480
  23:40:03 STEP 1080 H= 270:00 +CPU= 67.060
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001080
  23:40:56 STEP 1081 H= 270:15 +CPU= 53.217
  23:41:50 STEP 1082 H= 270:30 +CPU= 53.854
  23:42:44 STEP 1083 H= 270:45 +CPU= 53.937
  23:43:38 STEP 1084 H= 271:00 +CPU= 54.024
  23:44:32 STEP 1085 H= 271:15 +CPU= 53.560
  23:45:27 STEP 1086 H= 271:30 +CPU= 54.059
  23:46:21 STEP 1087 H= 271:45 +CPU= 53.331
  23:47:18 STEP 1088 H= 272:00 +CPU= 55.341
  23:48:12 STEP 1089 H= 272:15 +CPU= 53.802
  23:49:06 STEP 1090 H= 272:30 +CPU= 53.217
  23:50:00 STEP 1091 H= 272:45 +CPU= 53.334
  23:51:05 STEP 1092 H= 273:00 +CPU= 65.104
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001092
  23:51:59 STEP 1093 H= 273:15 +CPU= 53.910
  23:52:53 STEP 1094 H= 273:30 +CPU= 53.168
  23:53:48 STEP 1095 H= 273:45 +CPU= 54.377
  23:54:44 STEP 1096 H= 274:00 +CPU= 54.635
  23:55:37 STEP 1097 H= 274:15 +CPU= 53.309
  23:56:31 STEP 1098 H= 274:30 +CPU= 53.510
  23:57:25 STEP 1099 H= 274:45 +CPU= 53.931
  23:58:21 STEP 1100 H= 275:00 +CPU= 55.625
  23:59:15 STEP 1101 H= 275:15 +CPU= 53.823
  00:00:09 STEP 1102 H= 275:30 +CPU= 53.830
  00:01:03 STEP 1103 H= 275:45 +CPU= 54.379
  00:02:15 STEP 1104 H= 276:00 +CPU= 71.800
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/28/ICMUAh7zg+001104
  00:03:09 STEP 1105 H= 276:15 +CPU= 52.913
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4411MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4411 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.174] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000343:1717823023:52966.175] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/28/oifs_43r3_model.exe' : sigcontextptr=0x7fff1c7410d0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/28/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.184] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/28/oifs_43r3_model.exe' : sigcontextptr=0x7fff1c740fe0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/28/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.209] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:C04403UB14:1:1:3936318:3936318] [20240608:000344:1717823024:52967.209] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.657174249082187E-05 0.000000000000000E+00 0.743304302268798E-02
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.207548492185301E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.271471321959006E-05 0.000000000000000E+00 0.920037228216400E-02
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.150581838280638E-04 0.000000000000000E+00 0.158779219084829E-01
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.207096252156786E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.233402874692725E-04 0.000000000000000E+00 0.247639106502519E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.147721309436160E-04 0.000000000000000E+00 0.703995298731130E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.486182160942103E-05 0.000000000000000E+00 0.163230356024284E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.185157116629660E-05 0.000000000000000E+00 0.109278700127643E-01
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.128565125455655E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1104 STEPO   F00000000
 CREATE RESTART FILE ./srf00111200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00111200.0001 MODE=w
 IO-STREAM CLOSED - ./srf00111200.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    4.1383 SECONDS******** Mbytes/s, TOTAL TIME=    5.2971(78.1%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00110000.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1104 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001104 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001104
 MPI-TASK:   1 -    316111508 BYTES IN      6 RECORDS TRANSFERRED IN    0.0006 SECONDS526852.5 Mbytes/s, TOTAL TIME=    0.0183( 3.3%)
 NSTEP =  1104 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001104 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001104
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0154 SECONDS 45935.0 Mbytes/s, TOTAL TIME=    0.5898( 2.6%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001104 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001104
 MPI-TASK:   1 -   5057784128 BYTES IN    125 RECORDS TRANSFERRED IN    0.0103 SECONDS491047.0 Mbytes/s, TOTAL TIME=    0.3860( 2.7%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001104 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001104
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0003 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0074( 4.1%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001104 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001104
 MPI-TASK:   1 -    307531779 BYTES IN      4 RECORDS TRANSFERRED IN    0.0005 SECONDS615063.5 Mbytes/s, TOTAL TIME=    0.0118( 4.2%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1104 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1104 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001104 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001104
 MPI-TASK:   1 -    316111508 BYTES IN      7 RECORDS TRANSFERRED IN    0.0008 SECONDS395139.4 Mbytes/s, TOTAL TIME=    0.0206( 3.9%)
 00:02:15 STEP 1104 H= 276:00 +CPU= 71.800
ISTEP=    1105 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1105 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1105 STEPO   0AAA00AAA
 00:03:09 STEP 1105 H= 276:15 +CPU= 52.913
 NSTEP =  1106 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 00:01:09 0AA000000 STEPO     1104     5.934    5.934    5.959    874:59    880:12 0.32331902866552E-04       2GB       0MB
 00:01:12 FULLPOS-B DYNFPOS   1104     2.911    2.911    2.914    875:02    880:15 0.32331902866552E-04       2GB       0MB
 00:01:19 FULLPOS-V DYNFPOS   1104     6.234    6.234    6.238    875:08    880:21 0.32331902866552E-04       2GB       0MB
 00:01:20 FULLPOS-T DYNFPOS   1104     1.074    1.074    1.075    875:09    880:22 0.32331902866552E-04       2GB       0MB
 00:01:21 0AAA00AAA STEPO     1104     1.097    1.097    1.101    875:10    880:23 0.32331902866552E-04       2GB       0MB
 00:02:15 0AAA00AAA STEPO     1105    54.236   54.236   54.413    876:04    881:17 0.32331902866552E-04       2GB       0MB
 00:03:09 0AAA00AAA STEPO     1106    52.913   52.913   53.742    876:57    882:11 0.32331902866552E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1104",
 CTIME   = "00111200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1092, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/28/progress_file_12284102.xml

<running_values>
  <last_cpu_time>52644.640000</last_cpu_time>
  <upload_file_number>11</upload_file_number>
  <last_iter>1106</last_iter>
  <last_upload>950400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 03:00:10 1538899 22435682 oifs_43r3_bl_a1r7_2016092300_20_1017_12284102_0 864,000 44,655 0.0517


©2024 climateprediction.net