climateprediction.net home page
Task 22444963

Task 22444963

Name oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0
Workunit 12291363
Created 13 Jun 2024, 0:26:03 UTC
Sent 14 Jun 2024, 0:04:49 UTC
Report deadline 13 Aug 2024, 0:04:49 UTC
Received 14 Jun 2024, 13:03:54 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538903
Run time 12 hours 56 min 13 sec
CPU time 12 hours 49 min 59 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 3.86 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.45 MB
Peak swap size 5,981.13 MB
Peak disk usage 1,286.82 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0_r559677836_6.zip
Uploading the intermediate file: upload_file_6.zip
  02:54:36 STEP  673 H= 168:15 +CPU= 46.620
  02:55:23 STEP  674 H= 168:30 +CPU= 47.006
  02:56:10 STEP  675 H= 168:45 +CPU= 46.647
  02:56:57 STEP  676 H= 169:00 +CPU= 46.343
  02:57:44 STEP  677 H= 169:15 +CPU= 46.755
  02:58:31 STEP  678 H= 169:30 +CPU= 46.459
  02:59:17 STEP  679 H= 169:45 +CPU= 45.565
  03:00:07 STEP  680 H= 170:00 +CPU= 47.826
  03:00:54 STEP  681 H= 170:15 +CPU= 47.289
  03:01:39 STEP  682 H= 170:30 +CPU= 44.282
  03:02:27 STEP  683 H= 170:45 +CPU= 47.049
  03:03:26 STEP  684 H= 171:00 +CPU= 59.087
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000684
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000684
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000684
  03:04:12 STEP  685 H= 171:15 +CPU= 45.682
  03:04:55 STEP  686 H= 171:30 +CPU= 43.699
  03:05:42 STEP  687 H= 171:45 +CPU= 46.041
  03:06:30 STEP  688 H= 172:00 +CPU= 47.040
  03:07:17 STEP  689 H= 172:15 +CPU= 47.377
  03:08:05 STEP  690 H= 172:30 +CPU= 47.682
  03:08:52 STEP  691 H= 172:45 +CPU= 47.389
  03:09:41 STEP  692 H= 173:00 +CPU= 48.534
  03:10:29 STEP  693 H= 173:15 +CPU= 47.503
  03:11:12 STEP  694 H= 173:30 +CPU= 43.694
  03:12:01 STEP  695 H= 173:45 +CPU= 48.290
  03:13:01 STEP  696 H= 174:00 +CPU= 60.164
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000696
  03:13:50 STEP  697 H= 174:15 +CPU= 47.743
  03:14:38 STEP  698 H= 174:30 +CPU= 47.855
  03:15:25 STEP  699 H= 174:45 +CPU= 47.325
  03:16:12 STEP  700 H= 175:00 +CPU= 46.296
  03:16:58 STEP  701 H= 175:15 +CPU= 46.355
  03:17:45 STEP  702 H= 175:30 +CPU= 47.088
  03:18:32 STEP  703 H= 175:45 +CPU= 46.399
  03:19:23 STEP  704 H= 176:00 +CPU= 48.763
  03:20:10 STEP  705 H= 176:15 +CPU= 46.884
  03:20:56 STEP  706 H= 176:30 +CPU= 44.763
  03:21:43 STEP  707 H= 176:45 +CPU= 46.484
  03:22:41 STEP  708 H= 177:00 +CPU= 57.625
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000708
  03:23:29 STEP  709 H= 177:15 +CPU= 47.765
  03:24:17 STEP  710 H= 177:30 +CPU= 47.951
  03:25:05 STEP  711 H= 177:45 +CPU= 47.325
  03:25:49 STEP  712 H= 178:00 +CPU= 44.457
  03:26:37 STEP  713 H= 178:15 +CPU= 48.100
  03:27:23 STEP  714 H= 178:30 +CPU= 45.427
  03:28:11 STEP  715 H= 178:45 +CPU= 46.948
  03:28:59 STEP  716 H= 179:00 +CPU= 47.828
  03:29:45 STEP  717 H= 179:15 +CPU= 45.010
  03:30:29 STEP  718 H= 179:30 +CPU= 44.657
  03:31:16 STEP  719 H= 179:45 +CPU= 46.694
  03:32:19 STEP  720 H= 180:00 +CPU= 63.254
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000720
  03:33:07 STEP  721 H= 180:15 +CPU= 47.070
  03:33:54 STEP  722 H= 180:30 +CPU= 47.196
  03:34:41 STEP  723 H= 180:45 +CPU= 46.654
  03:35:27 STEP  724 H= 181:00 +CPU= 46.408
  03:36:13 STEP  725 H= 181:15 +CPU= 45.286
  03:37:00 STEP  726 H= 181:30 +CPU= 46.707
  03:37:48 STEP  727 H= 181:45 +CPU= 47.114
  03:38:37 STEP  728 H= 182:00 +CPU= 49.084
  03:39:25 STEP  729 H= 182:15 +CPU= 47.767
  03:40:12 STEP  730 H= 182:30 +CPU= 46.222
  03:41:00 STEP  731 H= 182:45 +CPU= 47.860
  03:42:00 STEP  732 H= 183:00 +CPU= 59.797
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000732
  03:42:48 STEP  733 H= 183:15 +CPU= 47.634
  03:43:35 STEP  734 H= 183:30 +CPU= 46.787
  03:44:24 STEP  735 H= 183:45 +CPU= 46.133
  03:45:12 STEP  736 H= 184:00 +CPU= 47.216
  03:45:55 STEP  737 H= 184:15 +CPU= 42.979
  03:46:42 STEP  738 H= 184:30 +CPU= 46.875
  03:47:29 STEP  739 H= 184:45 +CPU= 47.368
  03:48:17 STEP  740 H= 185:00 +CPU= 48.114
  03:49:06 STEP  741 H= 185:15 +CPU= 48.216
  03:49:51 STEP  742 H= 185:30 +CPU= 45.494
  03:50:39 STEP  743 H= 185:45 +CPU= 47.449
  03:51:35 STEP  744 H= 186:00 +CPU= 55.360
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000744
  03:52:22 STEP  745 H= 186:15 +CPU= 47.431
  03:53:10 STEP  746 H= 186:30 +CPU= 47.975
  03:53:57 STEP  747 H= 186:45 +CPU= 46.993
  03:54:45 STEP  748 H= 187:00 +CPU= 48.132
  03:55:31 STEP  749 H= 187:15 +CPU= 44.947
  03:56:15 STEP  750 H= 187:30 +CPU= 44.374
  03:57:03 STEP  751 H= 187:45 +CPU= 47.162
  03:57:50 STEP  752 H= 188:00 +CPU= 47.142
  03:58:38 STEP  753 H= 188:15 +CPU= 47.994
  03:59:26 STEP  754 H= 188:30 +CPU= 47.117
  04:00:14 STEP  755 H= 188:45 +CPU= 46.891
  04:01:12 STEP  756 H= 189:00 +CPU= 57.439
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000756
  04:02:00 STEP  757 H= 189:15 +CPU= 47.635
  04:02:47 STEP  758 H= 189:30 +CPU= 47.049
  04:03:34 STEP  759 H= 189:45 +CPU= 47.067
  04:04:22 STEP  760 H= 190:00 +CPU= 46.385
  04:05:10 STEP  761 H= 190:15 +CPU= 46.916
  04:05:58 STEP  762 H= 190:30 +CPU= 45.203
  04:06:45 STEP  763 H= 190:45 +CPU= 45.776
  04:07:33 STEP  764 H= 191:00 +CPU= 47.929
  04:08:22 STEP  765 H= 191:15 +CPU= 43.831
  04:09:06 STEP  766 H= 191:30 +CPU= 44.230
  04:09:52 STEP  767 H= 191:45 +CPU= 45.756
  04:10:51 STEP  768 H= 192:00 +CPU= 58.794
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0_r559677836_7.zip
Uploading the intermediate file: upload_file_7.zip
  04:11:37 STEP  769 H= 192:15 +CPU= 46.116
  04:12:22 STEP  770 H= 192:30 +CPU= 44.846
  04:13:09 STEP  771 H= 192:45 +CPU= 45.599
  04:13:55 STEP  772 H= 193:00 +CPU= 45.314
  04:14:40 STEP  773 H= 193:15 +CPU= 43.807
  04:15:23 STEP  774 H= 193:30 +CPU= 42.339
  04:16:09 STEP  775 H= 193:45 +CPU= 45.954
  04:16:57 STEP  776 H= 194:00 +CPU= 47.827
  04:17:44 STEP  777 H= 194:15 +CPU= 46.921
  04:18:31 STEP  778 H= 194:30 +CPU= 46.301
  04:19:18 STEP  779 H= 194:45 +CPU= 47.013
  04:20:14 STEP  780 H= 195:00 +CPU= 56.313
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000780
  04:20:58 STEP  781 H= 195:15 +CPU= 43.699
  04:21:43 STEP  782 H= 195:30 +CPU= 44.112
  04:22:27 STEP  783 H= 195:45 +CPU= 44.437
  04:23:14 STEP  784 H= 196:00 +CPU= 46.955
  04:23:59 STEP  785 H= 196:15 +CPU= 44.840
  04:24:46 STEP  786 H= 196:30 +CPU= 46.286
  04:25:33 STEP  787 H= 196:45 +CPU= 46.715
  04:26:19 STEP  788 H= 197:00 +CPU= 45.611
  04:27:05 STEP  789 H= 197:15 +CPU= 46.672
  04:27:52 STEP  790 H= 197:30 +CPU= 46.890
  04:28:40 STEP  791 H= 197:45 +CPU= 46.810
  04:29:33 STEP  792 H= 198:00 +CPU= 53.092
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000792
  04:30:17 STEP  793 H= 198:15 +CPU= 43.527
  04:31:03 STEP  794 H= 198:30 +CPU= 46.414
  04:31:50 STEP  795 H= 198:45 +CPU= 46.747
  04:32:36 STEP  796 H= 199:00 +CPU= 45.186
  04:33:22 STEP  797 H= 199:15 +CPU= 45.809
  04:34:06 STEP  798 H= 199:30 +CPU= 44.074
  04:34:51 STEP  799 H= 199:45 +CPU= 44.207
  04:35:36 STEP  800 H= 200:00 +CPU= 44.675
  04:36:21 STEP  801 H= 200:15 +CPU= 44.048
  04:37:07 STEP  802 H= 200:30 +CPU= 46.312
  04:37:53 STEP  803 H= 200:45 +CPU= 45.637
  04:38:46 STEP  804 H= 201:00 +CPU= 52.864
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000804
  04:39:33 STEP  805 H= 201:15 +CPU= 46.909
  04:40:20 STEP  806 H= 201:30 +CPU= 45.730
  04:41:05 STEP  807 H= 201:45 +CPU= 44.218
  04:41:52 STEP  808 H= 202:00 +CPU= 46.798
  04:42:40 STEP  809 H= 202:15 +CPU= 47.578
  04:43:28 STEP  810 H= 202:30 +CPU= 46.952
  04:44:15 STEP  811 H= 202:45 +CPU= 47.489
  04:45:02 STEP  812 H= 203:00 +CPU= 46.269
  04:45:49 STEP  813 H= 203:15 +CPU= 47.623
  04:46:37 STEP  814 H= 203:30 +CPU= 47.898
  04:47:24 STEP  815 H= 203:45 +CPU= 46.543
  04:48:28 STEP  816 H= 204:00 +CPU= 63.860
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000816
  04:49:13 STEP  817 H= 204:15 +CPU= 44.150
  04:50:00 STEP  818 H= 204:30 +CPU= 46.853
  04:50:47 STEP  819 H= 204:45 +CPU= 46.497
  04:51:33 STEP  820 H= 205:00 +CPU= 45.404
  04:52:18 STEP  821 H= 205:15 +CPU= 44.428
  04:53:03 STEP  822 H= 205:30 +CPU= 44.958
  04:53:47 STEP  823 H= 205:45 +CPU= 44.402
  04:54:30 STEP  824 H= 206:00 +CPU= 43.381
  04:55:13 STEP  825 H= 206:15 +CPU= 42.428
  04:55:57 STEP  826 H= 206:30 +CPU= 43.775
  04:56:45 STEP  827 H= 206:45 +CPU= 47.644
  04:57:44 STEP  828 H= 207:00 +CPU= 58.584
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000828
  04:58:31 STEP  829 H= 207:15 +CPU= 46.445
  04:59:17 STEP  830 H= 207:30 +CPU= 45.523
  05:00:03 STEP  831 H= 207:45 +CPU= 46.585
  05:00:51 STEP  832 H= 208:00 +CPU= 47.657
  05:01:38 STEP  833 H= 208:15 +CPU= 46.869
  05:02:25 STEP  834 H= 208:30 +CPU= 46.608
  05:03:13 STEP  835 H= 208:45 +CPU= 48.164
  05:04:00 STEP  836 H= 209:00 +CPU= 46.674
  05:04:44 STEP  837 H= 209:15 +CPU= 43.852
  05:05:30 STEP  838 H= 209:30 +CPU= 45.839
  05:06:17 STEP  839 H= 209:45 +CPU= 47.193
  05:07:16 STEP  840 H= 210:00 +CPU= 58.265
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000840
  05:08:02 STEP  841 H= 210:15 +CPU= 45.784
  05:08:45 STEP  842 H= 210:30 +CPU= 42.443
  05:09:30 STEP  843 H= 210:45 +CPU= 44.723
  05:10:18 STEP  844 H= 211:00 +CPU= 47.840
  05:11:05 STEP  845 H= 211:15 +CPU= 46.249
  05:11:51 STEP  846 H= 211:30 +CPU= 45.672
  05:12:36 STEP  847 H= 211:45 +CPU= 45.200
  05:13:23 STEP  848 H= 212:00 +CPU= 45.235
  05:14:10 STEP  849 H= 212:15 +CPU= 46.810
  05:14:58 STEP  850 H= 212:30 +CPU= 47.902
  05:15:46 STEP  851 H= 212:45 +CPU= 47.890
  05:16:44 STEP  852 H= 213:00 +CPU= 57.195
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000852
  05:17:27 STEP  853 H= 213:15 +CPU= 43.349
  05:18:15 STEP  854 H= 213:30 +CPU= 45.904
  05:18:58 STEP  855 H= 213:45 +CPU= 43.229
  05:19:48 STEP  856 H= 214:00 +CPU= 47.356
  05:20:36 STEP  857 H= 214:15 +CPU= 46.879
  05:21:23 STEP  858 H= 214:30 +CPU= 45.642
  05:22:10 STEP  859 H= 214:45 +CPU= 46.737
  05:22:57 STEP  860 H= 215:00 +CPU= 46.780
  05:23:43 STEP  861 H= 215:15 +CPU= 45.723
  05:24:30 STEP  862 H= 215:30 +CPU= 47.436
  05:25:18 STEP  863 H= 215:45 +CPU= 46.669
  05:26:23 STEP  864 H= 216:00 +CPU= 63.801
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0_r559677836_8.zip
Uploading the intermediate file: upload_file_8.zip
  05:27:09 STEP  865 H= 216:15 +CPU= 44.573
  05:27:53 STEP  866 H= 216:30 +CPU= 44.567
  05:28:41 STEP  867 H= 216:45 +CPU= 47.640
  05:29:31 STEP  868 H= 217:00 +CPU= 48.830
  05:30:19 STEP  869 H= 217:15 +CPU= 48.208
  05:31:07 STEP  870 H= 217:30 +CPU= 47.389
  05:31:56 STEP  871 H= 217:45 +CPU= 48.749
  05:32:46 STEP  872 H= 218:00 +CPU= 49.611
  05:33:32 STEP  873 H= 218:15 +CPU= 46.196
  05:34:18 STEP  874 H= 218:30 +CPU= 45.967
  05:35:06 STEP  875 H= 218:45 +CPU= 47.739
  05:36:05 STEP  876 H= 219:00 +CPU= 58.479
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000876
  05:36:53 STEP  877 H= 219:15 +CPU= 47.537
  05:37:39 STEP  878 H= 219:30 +CPU= 46.020
  05:38:28 STEP  879 H= 219:45 +CPU= 48.301
  05:39:17 STEP  880 H= 220:00 +CPU= 48.948
  05:40:01 STEP  881 H= 220:15 +CPU= 44.083
  05:40:44 STEP  882 H= 220:30 +CPU= 42.588
  05:41:31 STEP  883 H= 220:45 +CPU= 46.855
  05:42:19 STEP  884 H= 221:00 +CPU= 48.550
  05:43:08 STEP  885 H= 221:15 +CPU= 48.306
  05:43:56 STEP  886 H= 221:30 +CPU= 47.975
  05:44:42 STEP  887 H= 221:45 +CPU= 45.628
  05:45:43 STEP  888 H= 222:00 +CPU= 60.843
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000888
  05:46:31 STEP  889 H= 222:15 +CPU= 47.854
  05:47:17 STEP  890 H= 222:30 +CPU= 46.219
  05:48:03 STEP  891 H= 222:45 +CPU= 46.172
  05:48:48 STEP  892 H= 223:00 +CPU= 44.339
  05:49:34 STEP  893 H= 223:15 +CPU= 45.730
  05:50:21 STEP  894 H= 223:30 +CPU= 47.114
  05:51:09 STEP  895 H= 223:45 +CPU= 47.118
  05:51:59 STEP  896 H= 224:00 +CPU= 49.326
  05:52:46 STEP  897 H= 224:15 +CPU= 47.129
  05:53:32 STEP  898 H= 224:30 +CPU= 45.741
  05:54:15 STEP  899 H= 224:45 +CPU= 43.751
  05:55:15 STEP  900 H= 225:00 +CPU= 59.125
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000900
  05:56:02 STEP  901 H= 225:15 +CPU= 47.445
  05:56:50 STEP  902 H= 225:30 +CPU= 47.142
  05:57:37 STEP  903 H= 225:45 +CPU= 47.673
  05:58:27 STEP  904 H= 226:00 +CPU= 49.241
  05:59:15 STEP  905 H= 226:15 +CPU= 48.032
  06:00:02 STEP  906 H= 226:30 +CPU= 46.491
  06:00:46 STEP  907 H= 226:45 +CPU= 44.296
  06:01:35 STEP  908 H= 227:00 +CPU= 47.897
  06:02:21 STEP  909 H= 227:15 +CPU= 45.771
  06:03:09 STEP  910 H= 227:30 +CPU= 48.286
  06:03:54 STEP  911 H= 227:45 +CPU= 44.592
  06:05:00 STEP  912 H= 228:00 +CPU= 65.518
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000912
  06:05:47 STEP  913 H= 228:15 +CPU= 46.746
  06:06:35 STEP  914 H= 228:30 +CPU= 47.077
  06:07:23 STEP  915 H= 228:45 +CPU= 47.460
  06:08:12 STEP  916 H= 229:00 +CPU= 48.610
  06:09:01 STEP  917 H= 229:15 +CPU= 48.214
  06:09:48 STEP  918 H= 229:30 +CPU= 47.308
  06:10:35 STEP  919 H= 229:45 +CPU= 46.862
  06:11:24 STEP  920 H= 230:00 +CPU= 48.106
  06:12:12 STEP  921 H= 230:15 +CPU= 48.337
  06:12:59 STEP  922 H= 230:30 +CPU= 46.904
  06:13:47 STEP  923 H= 230:45 +CPU= 47.829
  06:14:42 STEP  924 H= 231:00 +CPU= 55.093
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000924
  06:15:29 STEP  925 H= 231:15 +CPU= 46.956
  06:16:17 STEP  926 H= 231:30 +CPU= 47.116
  06:17:00 STEP  927 H= 231:45 +CPU= 43.419
  06:17:49 STEP  928 H= 232:00 +CPU= 48.509
  06:18:38 STEP  929 H= 232:15 +CPU= 48.462
  06:19:26 STEP  930 H= 232:30 +CPU= 48.398
  06:20:15 STEP  931 H= 232:45 +CPU= 48.675
  06:21:03 STEP  932 H= 233:00 +CPU= 47.689
  06:21:50 STEP  933 H= 233:15 +CPU= 46.352
  06:22:36 STEP  934 H= 233:30 +CPU= 45.935
  06:23:24 STEP  935 H= 233:45 +CPU= 47.746
  06:24:22 STEP  936 H= 234:00 +CPU= 57.780
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000936
  06:25:09 STEP  937 H= 234:15 +CPU= 45.190
  06:25:57 STEP  938 H= 234:30 +CPU= 47.506
  06:26:46 STEP  939 H= 234:45 +CPU= 48.167
  06:27:34 STEP  940 H= 235:00 +CPU= 48.521
  06:28:20 STEP  941 H= 235:15 +CPU= 45.169
  06:29:07 STEP  942 H= 235:30 +CPU= 47.048
  06:29:54 STEP  943 H= 235:45 +CPU= 46.603
  06:30:40 STEP  944 H= 236:00 +CPU= 45.687
  06:31:24 STEP  945 H= 236:15 +CPU= 43.949
  06:32:10 STEP  946 H= 236:30 +CPU= 45.245
  06:32:54 STEP  947 H= 236:45 +CPU= 43.616
  06:33:50 STEP  948 H= 237:00 +CPU= 54.670
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000948
  06:34:38 STEP  949 H= 237:15 +CPU= 46.736
  06:35:25 STEP  950 H= 237:30 +CPU= 46.598
  06:36:12 STEP  951 H= 237:45 +CPU= 46.222
  06:36:57 STEP  952 H= 238:00 +CPU= 44.550
  06:37:44 STEP  953 H= 238:15 +CPU= 47.510
  06:38:30 STEP  954 H= 238:30 +CPU= 45.399
  06:39:17 STEP  955 H= 238:45 +CPU= 46.403
  06:40:05 STEP  956 H= 239:00 +CPU= 48.413
  06:40:51 STEP  957 H= 239:15 +CPU= 45.650
  06:41:38 STEP  958 H= 239:30 +CPU= 46.713
  06:42:25 STEP  959 H= 239:45 +CPU= 47.270
  06:43:28 STEP  960 H= 240:00 +CPU= 62.363
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0_r559677836_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  06:44:16 STEP  961 H= 240:15 +CPU= 47.415
  06:45:04 STEP  962 H= 240:30 +CPU= 47.959
  06:45:51 STEP  963 H= 240:45 +CPU= 46.403
  06:46:38 STEP  964 H= 241:00 +CPU= 47.223
  06:47:23 STEP  965 H= 241:15 +CPU= 44.782
  06:48:10 STEP  966 H= 241:30 +CPU= 46.555
  06:48:55 STEP  967 H= 241:45 +CPU= 44.439
  06:49:43 STEP  968 H= 242:00 +CPU= 47.793
  06:50:29 STEP  969 H= 242:15 +CPU= 45.568
  06:51:15 STEP  970 H= 242:30 +CPU= 46.185
  06:51:59 STEP  971 H= 242:45 +CPU= 43.917
  06:52:53 STEP  972 H= 243:00 +CPU= 54.051
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000972
  06:53:36 STEP  973 H= 243:15 +CPU= 42.390
  06:54:18 STEP  974 H= 243:30 +CPU= 42.374
  06:55:03 STEP  975 H= 243:45 +CPU= 44.451
  06:55:51 STEP  976 H= 244:00 +CPU= 48.228
  06:56:38 STEP  977 H= 244:15 +CPU= 46.846
  06:57:25 STEP  978 H= 244:30 +CPU= 46.679
  06:58:12 STEP  979 H= 244:45 +CPU= 46.844
  06:58:57 STEP  980 H= 245:00 +CPU= 45.063
  06:59:44 STEP  981 H= 245:15 +CPU= 47.732
  07:00:32 STEP  982 H= 245:30 +CPU= 47.797
  07:01:20 STEP  983 H= 245:45 +CPU= 47.775
  07:02:19 STEP  984 H= 246:00 +CPU= 58.816
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000984
  07:03:07 STEP  985 H= 246:15 +CPU= 47.105
  07:03:51 STEP  986 H= 246:30 +CPU= 44.697
  07:04:39 STEP  987 H= 246:45 +CPU= 47.718
  07:05:25 STEP  988 H= 247:00 +CPU= 46.074
  07:06:09 STEP  989 H= 247:15 +CPU= 43.739
  07:06:55 STEP  990 H= 247:30 +CPU= 45.576
  07:07:43 STEP  991 H= 247:45 +CPU= 48.085
  07:08:29 STEP  992 H= 248:00 +CPU= 45.652
  07:09:14 STEP  993 H= 248:15 +CPU= 44.592
  07:10:04 STEP  994 H= 248:30 +CPU= 48.447
  07:10:52 STEP  995 H= 248:45 +CPU= 47.784
  07:11:50 STEP  996 H= 249:00 +CPU= 58.086
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+000996
  07:12:34 STEP  997 H= 249:15 +CPU= 43.793
  07:13:22 STEP  998 H= 249:30 +CPU= 47.745
  07:14:10 STEP  999 H= 249:45 +CPU= 48.402
  07:14:59 STEP 1000 H= 250:00 +CPU= 48.873
  07:15:43 STEP 1001 H= 250:15 +CPU= 43.769
  07:16:29 STEP 1002 H= 250:30 +CPU= 45.931
  07:17:16 STEP 1003 H= 250:45 +CPU= 47.186
  07:18:05 STEP 1004 H= 251:00 +CPU= 48.559
  07:18:51 STEP 1005 H= 251:15 +CPU= 45.888
  07:19:37 STEP 1006 H= 251:30 +CPU= 45.877
  07:20:25 STEP 1007 H= 251:45 +CPU= 47.748
  07:21:30 STEP 1008 H= 252:00 +CPU= 64.787
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+001008
  07:22:19 STEP 1009 H= 252:15 +CPU= 48.645
  07:23:08 STEP 1010 H= 252:30 +CPU= 48.399
  07:23:56 STEP 1011 H= 252:45 +CPU= 48.618
  07:24:42 STEP 1012 H= 253:00 +CPU= 45.009
  07:25:32 STEP 1013 H= 253:15 +CPU= 50.008
  07:26:20 STEP 1014 H= 253:30 +CPU= 47.538
  07:27:09 STEP 1015 H= 253:45 +CPU= 48.971
  07:27:54 STEP 1016 H= 254:00 +CPU= 45.067
  07:28:44 STEP 1017 H= 254:15 +CPU= 49.315
  07:29:32 STEP 1018 H= 254:30 +CPU= 47.914
  07:30:20 STEP 1019 H= 254:45 +CPU= 48.442
  07:31:18 STEP 1020 H= 255:00 +CPU= 57.408
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+001020
  07:32:04 STEP 1021 H= 255:15 +CPU= 46.176
  07:32:52 STEP 1022 H= 255:30 +CPU= 47.742
  07:33:40 STEP 1023 H= 255:45 +CPU= 47.403
  07:34:30 STEP 1024 H= 256:00 +CPU= 49.449
  07:35:19 STEP 1025 H= 256:15 +CPU= 48.953
  07:36:09 STEP 1026 H= 256:30 +CPU= 49.113
  07:36:54 STEP 1027 H= 256:45 +CPU= 44.944
  07:37:42 STEP 1028 H= 257:00 +CPU= 48.373
  07:38:30 STEP 1029 H= 257:15 +CPU= 46.278
  07:39:19 STEP 1030 H= 257:30 +CPU= 48.774
  07:40:08 STEP 1031 H= 257:45 +CPU= 48.833
  07:41:09 STEP 1032 H= 258:00 +CPU= 61.404
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+001032
  07:41:58 STEP 1033 H= 258:15 +CPU= 47.958
  07:42:46 STEP 1034 H= 258:30 +CPU= 48.326
  07:43:34 STEP 1035 H= 258:45 +CPU= 47.693
  07:44:23 STEP 1036 H= 259:00 +CPU= 46.944
  07:45:12 STEP 1037 H= 259:15 +CPU= 48.412
  07:45:56 STEP 1038 H= 259:30 +CPU= 44.355
  07:46:44 STEP 1039 H= 259:45 +CPU= 47.776
  07:47:30 STEP 1040 H= 260:00 +CPU= 46.340
  07:48:18 STEP 1041 H= 260:15 +CPU= 46.637
  07:49:06 STEP 1042 H= 260:30 +CPU= 46.698
  07:49:53 STEP 1043 H= 260:45 +CPU= 46.982
  07:50:50 STEP 1044 H= 261:00 +CPU= 56.265
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+001044
  07:51:37 STEP 1045 H= 261:15 +CPU= 46.619
  07:52:26 STEP 1046 H= 261:30 +CPU= 47.918
  07:53:16 STEP 1047 H= 261:45 +CPU= 48.913
  07:54:06 STEP 1048 H= 262:00 +CPU= 49.994
  07:54:55 STEP 1049 H= 262:15 +CPU= 49.409
  07:55:45 STEP 1050 H= 262:30 +CPU= 49.086
  07:56:34 STEP 1051 H= 262:45 +CPU= 48.187
  07:57:23 STEP 1052 H= 263:00 +CPU= 49.390
  07:58:10 STEP 1053 H= 263:15 +CPU= 47.229
  07:58:58 STEP 1054 H= 263:30 +CPU= 47.443
  07:59:41 STEP 1055 H= 263:45 +CPU= 43.535
  08:00:45 STEP 1056 H= 264:00 +CPU= 63.710
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/18/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12291363/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0_r559677836_10.zip
Uploading the intermediate file: upload_file_10.zip
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5026MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5026 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080115:1718370075:46496.288] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/18/oifs_43r3_model.exe' : sigcontextptr=0x7ffefbb7f590
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/18/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.288] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/18/oifs_43r3_model.exe' : sigcontextptr=0x7ffefbb7f4a0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/18/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.326] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:C04403UB01:1:1:932420:932420] [20240614:080116:1718370076:46497.326] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.383954059468733E-04 0.000000000000000E+00 0.196766577404883E-01
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.936452548573100E-04 0.000000000000000E+00 0.281256894094753E-01
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.692349368875201E-05 0.000000000000000E+00 0.108248030659272E-01
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.237879504661566E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.373709754300184E-05 0.000000000000000E+00 0.172017409681306E-01
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.370810797332634E-04 0.000000000000000E+00 0.196477526012092E-01
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.237634410549666E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.227286806746379E-04 0.000000000000000E+00 0.320137696338690E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.182896048764971E-04 0.000000000000000E+00 0.687280780181171E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.609565113585743E-05 0.000000000000000E+00 0.204696429654482E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.221533475438181E-05 0.000000000000000E+00 0.116108274502487E-01
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.222632839212224E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1056 STEPO   F00000000
 CREATE RESTART FILE ./srf00110000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00110000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00110000.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    3.3603 SECONDS******** Mbytes/s, TOTAL TIME=    4.2785(78.5%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00101200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1056 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001056 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001056
 MPI-TASK:   1 -    327275364 BYTES IN      6 RECORDS TRANSFERRED IN    0.0005 SECONDS654550.7 Mbytes/s, TOTAL TIME=    0.0158( 3.2%)
 NSTEP =  1056 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001056 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001056
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0129 SECONDS 54837.2 Mbytes/s, TOTAL TIME=    0.4703( 2.7%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001056 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001056
 MPI-TASK:   1 -   5236405824 BYTES IN    125 RECORDS TRANSFERRED IN    0.0093 SECONDS563054.4 Mbytes/s, TOTAL TIME=    0.3585( 2.6%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001056 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001056
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0005 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0065( 7.7%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001056 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001056
 MPI-TASK:   1 -    318695635 BYTES IN      4 RECORDS TRANSFERRED IN    0.0004 SECONDS796739.0 Mbytes/s, TOTAL TIME=    0.0104( 3.8%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1056 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1056 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001056 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001056
 MPI-TASK:   1 -    327275364 BYTES IN      7 RECORDS TRANSFERRED IN    0.0009 SECONDS363639.3 Mbytes/s, TOTAL TIME=    0.0198( 4.5%)
 08:00:45 STEP 1056 H= 264:00 +CPU= 63.710
 NSTEP =  1057 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 07:59:42 F00000000 STEPO     1056    43.730   43.730   43.753    768:26    773:23 0.43630548191976E-04       2GB       0MB
 07:59:46 0AA000000 STEPO     1056     4.830    4.830    4.837    768:31    773:27 0.43630548191976E-04       2GB       0MB
 07:59:49 FULLPOS-B DYNFPOS   1056     2.465    2.465    2.465    768:33    773:30 0.43630548191976E-04       2GB       0MB
 07:59:54 FULLPOS-V DYNFPOS   1056     5.127    5.127    5.127    768:38    773:35 0.43630548191976E-04       2GB       0MB
 07:59:55 FULLPOS-T DYNFPOS   1056     0.876    0.876    0.876    768:39    773:36 0.43630548191976E-04       2GB       0MB
 07:59:56 0AAA00AAA STEPO     1056     0.842    0.842    0.842    768:40    773:37 0.43630548191976E-04       2GB       0MB
 08:00:45 0AAA00AAA STEPO     1057    49.376   49.376   49.636    769:29    774:26 0.43630548191976E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1056",
 CTIME   = "00110000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1044, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/18/progress_file_12291363.xml

<running_values>
  <last_cpu_time>46178.240000</last_cpu_time>
  <upload_file_number>11</upload_file_number>
  <last_iter>1057</last_iter>
  <last_upload>950400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Jun 2024 12:02:21 1538903 22444963 oifs_43r3_bl_a2cc_2016092300_20_1018_12291363_0 864,000 41,559 0.0481


©2024 cpdn.org