climateprediction.net home page
Task 22449120

Task 22449120

Name oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1
Workunit 12294554
Created 15 Jun 2024, 12:35:51 UTC
Sent 15 Jun 2024, 12:36:00 UTC
Report deadline 14 Aug 2024, 12:36:00 UTC
Received 16 Jun 2024, 0:12:42 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1551095
Run time 9 hours 13 min 31 sec
CPU time 9 hours 9 min 9 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.35 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,362.20 MB
Stderr
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
00732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1_r1423833186_7.zip
Uploading the intermediate file: upload_file_7.zip
  19:32:17 STEP  769 H= 192:15 +CPU= 31.138
  19:32:46 STEP  770 H= 192:30 +CPU= 28.176
  19:33:14 STEP  771 H= 192:45 +CPU= 28.225
  19:33:43 STEP  772 H= 193:00 +CPU= 28.461
  19:34:12 STEP  773 H= 193:15 +CPU= 29.066
  19:34:40 STEP  774 H= 193:30 +CPU= 27.828
  19:35:08 STEP  775 H= 193:45 +CPU= 27.849
  19:35:38 STEP  776 H= 194:00 +CPU= 29.701
  19:36:06 STEP  777 H= 194:15 +CPU= 28.133
  19:36:34 STEP  778 H= 194:30 +CPU= 27.890
  19:37:02 STEP  779 H= 194:45 +CPU= 27.883
  19:37:38 STEP  780 H= 195:00 +CPU= 35.884
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000780
  19:38:06 STEP  781 H= 195:15 +CPU= 28.232
  19:38:34 STEP  782 H= 195:30 +CPU= 27.734
  19:39:02 STEP  783 H= 195:45 +CPU= 28.069
  19:39:32 STEP  784 H= 196:00 +CPU= 29.408
  19:40:00 STEP  785 H= 196:15 +CPU= 27.692
  19:40:29 STEP  786 H= 196:30 +CPU= 29.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:41:18 STEP  787 H= 196:45 +CPU= 27.186
  19:41:48 STEP  788 H= 197:00 +CPU= 29.533
  19:42:16 STEP  789 H= 197:15 +CPU= 28.073
  19:42:44 STEP  790 H= 197:30 +CPU= 28.399
  19:43:13 STEP  791 H= 197:45 +CPU= 28.388
  19:43:49 STEP  792 H= 198:00 +CPU= 35.679
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000792
  19:44:18 STEP  793 H= 198:15 +CPU= 28.830
  19:44:46 STEP  794 H= 198:30 +CPU= 28.052
  19:45:15 STEP  795 H= 198:45 +CPU= 28.922
  19:45:44 STEP  796 H= 199:00 +CPU= 28.558
  19:46:12 STEP  797 H= 199:15 +CPU= 28.239
  19:46:41 STEP  798 H= 199:30 +CPU= 28.938
  19:47:10 STEP  799 H= 199:45 +CPU= 28.420
  19:47:40 STEP  800 H= 200:00 +CPU= 29.995
  19:48:10 STEP  801 H= 200:15 +CPU= 29.548
  19:48:39 STEP  802 H= 200:30 +CPU= 28.648
  19:49:07 STEP  803 H= 200:45 +CPU= 28.535
  19:49:45 STEP  804 H= 201:00 +CPU= 37.621
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000804
  19:50:16 STEP  805 H= 201:15 +CPU= 30.244
  19:50:45 STEP  806 H= 201:30 +CPU= 29.072
  19:51:14 STEP  807 H= 201:45 +CPU= 28.958
  19:51:44 STEP  808 H= 202:00 +CPU= 29.797
  19:52:12 STEP  809 H= 202:15 +CPU= 27.723
  19:52:41 STEP  810 H= 202:30 +CPU= 28.886
  19:53:10 STEP  811 H= 202:45 +CPU= 29.045
  19:53:42 STEP  812 H= 203:00 +CPU= 30.923
  19:54:11 STEP  813 H= 203:15 +CPU= 29.572
  19:54:40 STEP  814 H= 203:30 +CPU= 28.735
  19:55:10 STEP  815 H= 203:45 +CPU= 29.738
  19:55:51 STEP  816 H= 204:00 +CPU= 41.190
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000816
  19:56:21 STEP  817 H= 204:15 +CPU= 29.645
  19:56:51 STEP  818 H= 204:30 +CPU= 30.039
  19:57:19 STEP  819 H= 204:45 +CPU= 28.004
  19:57:50 STEP  820 H= 205:00 +CPU= 30.156
  19:58:18 STEP  821 H= 205:15 +CPU= 28.462
  19:58:47 STEP  822 H= 205:30 +CPU= 28.588
  19:59:17 STEP  823 H= 205:45 +CPU= 30.095
  19:59:48 STEP  824 H= 206:00 +CPU= 30.599
  20:00:18 STEP  825 H= 206:15 +CPU= 29.605
  20:00:47 STEP  826 H= 206:30 +CPU= 28.855
  20:01:16 STEP  827 H= 206:45 +CPU= 28.353
  20:01:53 STEP  828 H= 207:00 +CPU= 37.056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000828
  20:02:23 STEP  829 H= 207:15 +CPU= 29.409
  20:02:51 STEP  830 H= 207:30 +CPU= 28.003
  20:03:21 STEP  831 H= 207:45 +CPU= 30.249
  20:03:52 STEP  832 H= 208:00 +CPU= 30.338
  20:04:21 STEP  833 H= 208:15 +CPU= 28.888
  20:04:51 STEP  834 H= 208:30 +CPU= 29.890
  20:05:21 STEP  835 H= 208:45 +CPU= 29.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:06:12 STEP  836 H= 209:00 +CPU= 29.515
  20:06:40 STEP  837 H= 209:15 +CPU= 28.523
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:07:30 STEP  838 H= 209:30 +CPU= 28.686
  20:08:00 STEP  839 H= 209:45 +CPU= 29.455
  20:08:38 STEP  840 H= 210:00 +CPU= 37.980
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000840
  20:09:09 STEP  841 H= 210:15 +CPU= 30.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:09:59 STEP  842 H= 210:30 +CPU= 28.810
  20:10:29 STEP  843 H= 210:45 +CPU= 30.144
  20:11:00 STEP  844 H= 211:00 +CPU= 30.589
  20:11:28 STEP  845 H= 211:15 +CPU= 28.762
  20:11:58 STEP  846 H= 211:30 +CPU= 29.440
  20:12:28 STEP  847 H= 211:45 +CPU= 29.457
  20:12:58 STEP  848 H= 212:00 +CPU= 29.709
  20:13:28 STEP  849 H= 212:15 +CPU= 30.027
  20:13:57 STEP  850 H= 212:30 +CPU= 28.910
  20:14:26 STEP  851 H= 212:45 +CPU= 28.861
  20:15:03 STEP  852 H= 213:00 +CPU= 37.027
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000852
  20:15:33 STEP  853 H= 213:15 +CPU= 29.656
  20:16:04 STEP  854 H= 213:30 +CPU= 30.176
  20:16:33 STEP  855 H= 213:45 +CPU= 29.279
  20:17:04 STEP  856 H= 214:00 +CPU= 30.546
  20:17:33 STEP  857 H= 214:15 +CPU= 29.398
  20:18:03 STEP  858 H= 214:30 +CPU= 29.453
  20:18:32 STEP  859 H= 214:45 +CPU= 28.853
  20:19:02 STEP  860 H= 215:00 +CPU= 29.490
  20:19:32 STEP  861 H= 215:15 +CPU= 29.898
  20:20:00 STEP  862 H= 215:30 +CPU= 28.104
  20:20:29 STEP  863 H= 215:45 +CPU= 29.264
  20:21:12 STEP  864 H= 216:00 +CPU= 42.105
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1_r1423833186_8.zip
Uploading the intermediate file: upload_file_8.zip
  20:21:43 STEP  865 H= 216:15 +CPU= 30.943
  20:22:13 STEP  866 H= 216:30 +CPU= 29.738
  20:22:43 STEP  867 H= 216:45 +CPU= 29.523
  20:23:13 STEP  868 H= 217:00 +CPU= 30.247
  20:23:45 STEP  869 H= 217:15 +CPU= 30.903
  20:24:15 STEP  870 H= 217:30 +CPU= 30.083
  20:24:45 STEP  871 H= 217:45 +CPU= 29.732
  20:25:16 STEP  872 H= 218:00 +CPU= 31.084
  20:25:45 STEP  873 H= 218:15 +CPU= 28.929
  20:26:16 STEP  874 H= 218:30 +CPU= 30.572
  20:26:45 STEP  875 H= 218:45 +CPU= 28.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:27:42 STEP  876 H= 219:00 +CPU= 36.168
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000876
  20:28:12 STEP  877 H= 219:15 +CPU= 29.703
  20:28:42 STEP  878 H= 219:30 +CPU= 29.361
  20:29:13 STEP  879 H= 219:45 +CPU= 30.674
  20:29:43 STEP  880 H= 220:00 +CPU= 29.922
  20:30:13 STEP  881 H= 220:15 +CPU= 30.202
  20:30:41 STEP  882 H= 220:30 +CPU= 28.180
  20:31:11 STEP  883 H= 220:45 +CPU= 29.593
  20:31:42 STEP  884 H= 221:00 +CPU= 30.658
  20:32:12 STEP  885 H= 221:15 +CPU= 30.563
  20:32:41 STEP  886 H= 221:30 +CPU= 28.902
  20:33:09 STEP  887 H= 221:45 +CPU= 28.078
  20:33:47 STEP  888 H= 222:00 +CPU= 37.461
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000888
  20:34:18 STEP  889 H= 222:15 +CPU= 30.180
  20:34:46 STEP  890 H= 222:30 +CPU= 28.460
  20:35:16 STEP  891 H= 222:45 +CPU= 29.521
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:36:07 STEP  892 H= 223:00 +CPU= 29.539
  20:36:35 STEP  893 H= 223:15 +CPU= 28.653
  20:37:05 STEP  894 H= 223:30 +CPU= 28.918
  20:37:34 STEP  895 H= 223:45 +CPU= 28.959
  20:38:04 STEP  896 H= 224:00 +CPU= 30.423
  20:38:34 STEP  897 H= 224:15 +CPU= 29.671
  20:39:04 STEP  898 H= 224:30 +CPU= 29.346
  20:39:32 STEP  899 H= 224:45 +CPU= 28.473
  20:40:10 STEP  900 H= 225:00 +CPU= 37.310
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000900
  20:40:40 STEP  901 H= 225:15 +CPU= 30.060
  20:41:10 STEP  902 H= 225:30 +CPU= 29.479
  20:41:38 STEP  903 H= 225:45 +CPU= 28.231
  20:42:09 STEP  904 H= 226:00 +CPU= 30.588
  20:42:37 STEP  905 H= 226:15 +CPU= 28.260
  20:43:07 STEP  906 H= 226:30 +CPU= 29.337
  20:43:36 STEP  907 H= 226:45 +CPU= 28.970
  20:44:06 STEP  908 H= 227:00 +CPU= 30.008
  20:44:35 STEP  909 H= 227:15 +CPU= 28.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:45:26 STEP  910 H= 227:30 +CPU= 28.908
  20:45:55 STEP  911 H= 227:45 +CPU= 28.855
  20:46:36 STEP  912 H= 228:00 +CPU= 40.724
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000912
  20:47:05 STEP  913 H= 228:15 +CPU= 29.496
  20:47:34 STEP  914 H= 228:30 +CPU= 28.904
  20:48:03 STEP  915 H= 228:45 +CPU= 28.471
  20:48:33 STEP  916 H= 229:00 +CPU= 30.118
  20:49:02 STEP  917 H= 229:15 +CPU= 28.755
  20:49:30 STEP  918 H= 229:30 +CPU= 28.079
  20:50:00 STEP  919 H= 229:45 +CPU= 29.520
  20:50:30 STEP  920 H= 230:00 +CPU= 30.246
  20:51:00 STEP  921 H= 230:15 +CPU= 28.932
  20:51:29 STEP  922 H= 230:30 +CPU= 29.247
  20:51:59 STEP  923 H= 230:45 +CPU= 29.364
  20:52:35 STEP  924 H= 231:00 +CPU= 36.437
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000924
  20:53:04 STEP  925 H= 231:15 +CPU= 28.546
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:13:47 STEP  926 H= 231:30 +CPU= 29.493
  21:14:16 STEP  927 H= 231:45 +CPU= 28.903
  21:14:46 STEP  928 H= 232:00 +CPU= 29.307
  21:15:16 STEP  929 H= 232:15 +CPU= 29.930
  21:15:44 STEP  930 H= 232:30 +CPU= 28.030
  21:16:14 STEP  931 H= 232:45 +CPU= 29.165
  21:16:45 STEP  932 H= 233:00 +CPU= 30.717
  21:17:16 STEP  933 H= 233:15 +CPU= 31.230
  21:17:45 STEP  934 H= 233:30 +CPU= 28.600
  21:18:14 STEP  935 H= 233:45 +CPU= 28.569
  21:18:51 STEP  936 H= 234:00 +CPU= 37.461
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000936
  21:19:20 STEP  937 H= 234:15 +CPU= 28.547
  21:19:49 STEP  938 H= 234:30 +CPU= 28.492
  21:20:18 STEP  939 H= 234:45 +CPU= 28.761
  21:20:48 STEP  940 H= 235:00 +CPU= 29.896
  21:21:18 STEP  941 H= 235:15 +CPU= 29.348
  21:21:46 STEP  942 H= 235:30 +CPU= 27.901
  21:22:15 STEP  943 H= 235:45 +CPU= 28.809
  21:22:45 STEP  944 H= 236:00 +CPU= 29.332
  21:23:14 STEP  945 H= 236:15 +CPU= 29.171
  21:23:43 STEP  946 H= 236:30 +CPU= 29.281
  21:24:13 STEP  947 H= 236:45 +CPU= 29.813
  21:24:50 STEP  948 H= 237:00 +CPU= 36.512
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000948
  21:25:19 STEP  949 H= 237:15 +CPU= 28.648
  21:25:48 STEP  950 H= 237:30 +CPU= 29.042
  21:26:19 STEP  951 H= 237:45 +CPU= 30.587
  21:26:49 STEP  952 H= 238:00 +CPU= 29.533
  21:27:19 STEP  953 H= 238:15 +CPU= 29.355
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:28:08 STEP  954 H= 238:30 +CPU= 28.192
  21:28:37 STEP  955 H= 238:45 +CPU= 28.791
  21:29:07 STEP  956 H= 239:00 +CPU= 29.808
  21:29:36 STEP  957 H= 239:15 +CPU= 28.720
  21:30:05 STEP  958 H= 239:30 +CPU= 29.373
  21:30:35 STEP  959 H= 239:45 +CPU= 29.448
  21:31:16 STEP  960 H= 240:00 +CPU= 41.169
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1_r1423833186_9.zip
Uploading the intermediate file: upload_file_9.zip
  21:31:48 STEP  961 H= 240:15 +CPU= 30.956
Uploading trickle at timestep: 864000
  21:32:18 STEP  962 H= 240:30 +CPU= 30.097
  21:32:47 STEP  963 H= 240:45 +CPU= 28.663
  21:33:17 STEP  964 H= 241:00 +CPU= 30.064
  21:33:47 STEP  965 H= 241:15 +CPU= 28.964
  21:34:16 STEP  966 H= 241:30 +CPU= 28.933
  21:34:45 STEP  967 H= 241:45 +CPU= 28.343
  21:35:14 STEP  968 H= 242:00 +CPU= 29.331
  21:35:44 STEP  969 H= 242:15 +CPU= 29.800
  21:36:13 STEP  970 H= 242:30 +CPU= 29.006
  21:36:42 STEP  971 H= 242:45 +CPU= 28.563
  21:37:19 STEP  972 H= 243:00 +CPU= 37.310
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000972
  21:37:50 STEP  973 H= 243:15 +CPU= 30.069
  21:38:19 STEP  974 H= 243:30 +CPU= 28.838
  21:38:47 STEP  975 H= 243:45 +CPU= 28.129
  21:39:17 STEP  976 H= 244:00 +CPU= 29.520
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:40:06 STEP  977 H= 244:15 +CPU= 28.030
  21:40:36 STEP  978 H= 244:30 +CPU= 29.457
  21:41:04 STEP  979 H= 244:45 +CPU= 28.582
  21:41:33 STEP  980 H= 245:00 +CPU= 28.920
  21:42:02 STEP  981 H= 245:15 +CPU= 28.523
  21:42:31 STEP  982 H= 245:30 +CPU= 28.817
  21:43:01 STEP  983 H= 245:45 +CPU= 29.378
  21:43:38 STEP  984 H= 246:00 +CPU= 36.440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000984
  21:44:07 STEP  985 H= 246:15 +CPU= 29.053
  21:44:37 STEP  986 H= 246:30 +CPU= 29.383
  21:45:06 STEP  987 H= 246:45 +CPU= 28.842
  21:45:36 STEP  988 H= 247:00 +CPU= 29.143
  21:46:05 STEP  989 H= 247:15 +CPU= 28.843
  21:46:34 STEP  990 H= 247:30 +CPU= 28.584
  21:47:03 STEP  991 H= 247:45 +CPU= 28.000
  21:47:32 STEP  992 H= 248:00 +CPU= 29.473
  21:48:01 STEP  993 H= 248:15 +CPU= 28.286
  21:48:29 STEP  994 H= 248:30 +CPU= 28.017
  21:48:58 STEP  995 H= 248:45 +CPU= 28.825
  21:49:36 STEP  996 H= 249:00 +CPU= 36.906
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+000996
  21:50:04 STEP  997 H= 249:15 +CPU= 28.295
  21:50:34 STEP  998 H= 249:30 +CPU= 29.283
  21:51:03 STEP  999 H= 249:45 +CPU= 29.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:51:54 STEP 1000 H= 250:00 +CPU= 29.213
  21:52:23 STEP 1001 H= 250:15 +CPU= 29.161
  21:52:52 STEP 1002 H= 250:30 +CPU= 29.122
  21:53:22 STEP 1003 H= 250:45 +CPU= 29.329
  21:53:53 STEP 1004 H= 251:00 +CPU= 30.226
  21:54:23 STEP 1005 H= 251:15 +CPU= 30.434
  21:54:53 STEP 1006 H= 251:30 +CPU= 29.705
  21:55:24 STEP 1007 H= 251:45 +CPU= 30.324
  21:56:06 STEP 1008 H= 252:00 +CPU= 41.589
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001008
  21:56:36 STEP 1009 H= 252:15 +CPU= 29.770
  21:57:04 STEP 1010 H= 252:30 +CPU= 27.979
  21:57:33 STEP 1011 H= 252:45 +CPU= 29.156
  21:58:03 STEP 1012 H= 253:00 +CPU= 29.527
  21:58:32 STEP 1013 H= 253:15 +CPU= 29.058
  21:59:02 STEP 1014 H= 253:30 +CPU= 29.410
  21:59:31 STEP 1015 H= 253:45 +CPU= 28.334
  22:00:03 STEP 1016 H= 254:00 +CPU= 32.015
  22:00:37 STEP 1017 H= 254:15 +CPU= 33.413
  22:01:07 STEP 1018 H= 254:30 +CPU= 29.644
  22:01:38 STEP 1019 H= 254:45 +CPU= 29.754
  22:02:15 STEP 1020 H= 255:00 +CPU= 36.956
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001020
  22:02:44 STEP 1021 H= 255:15 +CPU= 28.419
  22:03:12 STEP 1022 H= 255:30 +CPU= 28.260
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:04:03 STEP 1023 H= 255:45 +CPU= 29.280
  22:04:34 STEP 1024 H= 256:00 +CPU= 31.110
  22:05:04 STEP 1025 H= 256:15 +CPU= 30.103
  22:05:35 STEP 1026 H= 256:30 +CPU= 31.098
  22:06:06 STEP 1027 H= 256:45 +CPU= 30.481
  22:06:37 STEP 1028 H= 257:00 +CPU= 30.746
  22:07:08 STEP 1029 H= 257:15 +CPU= 31.025
  22:07:40 STEP 1030 H= 257:30 +CPU= 31.022
  22:08:11 STEP 1031 H= 257:45 +CPU= 31.643
  22:08:52 STEP 1032 H= 258:00 +CPU= 39.515
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001032
  22:09:22 STEP 1033 H= 258:15 +CPU= 30.562
  22:09:54 STEP 1034 H= 258:30 +CPU= 31.104
  22:10:24 STEP 1035 H= 258:45 +CPU= 30.225
  22:10:54 STEP 1036 H= 259:00 +CPU= 29.089
  22:11:24 STEP 1037 H= 259:15 +CPU= 30.020
  22:11:54 STEP 1038 H= 259:30 +CPU= 29.822
  22:12:24 STEP 1039 H= 259:45 +CPU= 29.589
  22:12:55 STEP 1040 H= 260:00 +CPU= 31.232
  22:13:26 STEP 1041 H= 260:15 +CPU= 30.632
  22:13:58 STEP 1042 H= 260:30 +CPU= 31.187
  22:14:28 STEP 1043 H= 260:45 +CPU= 30.583
  22:15:08 STEP 1044 H= 261:00 +CPU= 39.258
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001044
  22:15:38 STEP 1045 H= 261:15 +CPU= 30.624
  22:16:09 STEP 1046 H= 261:30 +CPU= 30.717
  22:16:40 STEP 1047 H= 261:45 +CPU= 31.064
  22:17:14 STEP 1048 H= 262:00 +CPU= 33.158
  22:17:45 STEP 1049 H= 262:15 +CPU= 31.105
  22:18:15 STEP 1050 H= 262:30 +CPU= 29.676
  22:18:45 STEP 1051 H= 262:45 +CPU= 29.942
  22:19:17 STEP 1052 H= 263:00 +CPU= 30.988
  22:19:47 STEP 1053 H= 263:15 +CPU= 29.614
  22:20:18 STEP 1054 H= 263:30 +CPU= 30.957
  22:20:48 STEP 1055 H= 263:45 +CPU= 30.387
  22:21:31 STEP 1056 H= 264:00 +CPU= 42.549
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1_r1423833186_10.zip
Uploading the intermediate file: upload_file_10.zip
  22:22:03 STEP 1057 H= 264:15 +CPU= 31.389
  22:22:33 STEP 1058 H= 264:30 +CPU= 29.603
  22:23:03 STEP 1059 H= 264:45 +CPU= 29.627
  22:23:33 STEP 1060 H= 265:00 +CPU= 30.131
  22:24:04 STEP 1061 H= 265:15 +CPU= 29.643
  22:24:34 STEP 1062 H= 265:30 +CPU= 30.119
  22:25:04 STEP 1063 H= 265:45 +CPU= 29.865
  22:25:35 STEP 1064 H= 266:00 +CPU= 30.670
  22:26:05 STEP 1065 H= 266:15 +CPU= 29.837
  22:26:34 STEP 1066 H= 266:30 +CPU= 28.894
  22:27:03 STEP 1067 H= 266:45 +CPU= 28.972
  22:27:40 STEP 1068 H= 267:00 +CPU= 36.674
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001068
  22:28:09 STEP 1069 H= 267:15 +CPU= 29.434
  22:28:40 STEP 1070 H= 267:30 +CPU= 30.208
  22:29:09 STEP 1071 H= 267:45 +CPU= 28.982
  22:29:39 STEP 1072 H= 268:00 +CPU= 29.361
  22:30:08 STEP 1073 H= 268:15 +CPU= 29.440
  22:30:38 STEP 1074 H= 268:30 +CPU= 29.375
  22:31:07 STEP 1075 H= 268:45 +CPU= 29.118
  22:31:38 STEP 1076 H= 269:00 +CPU= 29.718
  22:32:06 STEP 1077 H= 269:15 +CPU= 28.333
  22:32:36 STEP 1078 H= 269:30 +CPU= 28.777
  22:33:05 STEP 1079 H= 269:45 +CPU= 29.065
  22:33:42 STEP 1080 H= 270:00 +CPU= 37.069
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001080
  22:34:11 STEP 1081 H= 270:15 +CPU= 28.174
  22:34:40 STEP 1082 H= 270:30 +CPU= 29.191
  22:35:10 STEP 1083 H= 270:45 +CPU= 29.726
  22:35:41 STEP 1084 H= 271:00 +CPU= 30.254
  22:36:10 STEP 1085 H= 271:15 +CPU= 28.909
  22:36:39 STEP 1086 H= 271:30 +CPU= 28.529
  22:37:07 STEP 1087 H= 271:45 +CPU= 28.540
  22:37:37 STEP 1088 H= 272:00 +CPU= 29.477
  22:38:07 STEP 1089 H= 272:15 +CPU= 29.155
  22:38:36 STEP 1090 H= 272:30 +CPU= 28.739
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:39:26 STEP 1091 H= 272:45 +CPU= 28.577
  22:40:02 STEP 1092 H= 273:00 +CPU= 35.919
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001092
  22:40:31 STEP 1093 H= 273:15 +CPU= 28.863
  22:41:01 STEP 1094 H= 273:30 +CPU= 29.300
  22:41:31 STEP 1095 H= 273:45 +CPU= 29.832
  22:42:03 STEP 1096 H= 274:00 +CPU= 31.366
  22:42:32 STEP 1097 H= 274:15 +CPU= 29.136
  22:43:01 STEP 1098 H= 274:30 +CPU= 28.425
  22:43:30 STEP 1099 H= 274:45 +CPU= 28.657
  22:43:59 STEP 1100 H= 275:00 +CPU= 29.621
  22:44:29 STEP 1101 H= 275:15 +CPU= 29.566
  22:44:58 STEP 1102 H= 275:30 +CPU= 28.560
  22:45:28 STEP 1103 H= 275:45 +CPU= 29.136
  22:46:08 STEP 1104 H= 276:00 +CPU= 40.279
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001104
  22:46:37 STEP 1105 H= 276:15 +CPU= 28.797
  22:47:06 STEP 1106 H= 276:30 +CPU= 28.784
  22:47:35 STEP 1107 H= 276:45 +CPU= 28.130
  22:48:05 STEP 1108 H= 277:00 +CPU= 29.817
  22:48:34 STEP 1109 H= 277:15 +CPU= 29.027
  22:49:03 STEP 1110 H= 277:30 +CPU= 28.887
  22:49:32 STEP 1111 H= 277:45 +CPU= 28.865
  22:50:02 STEP 1112 H= 278:00 +CPU= 29.640
  22:50:31 STEP 1113 H= 278:15 +CPU= 28.685
  22:51:00 STEP 1114 H= 278:30 +CPU= 28.704
  22:51:29 STEP 1115 H= 278:45 +CPU= 29.328
  22:52:06 STEP 1116 H= 279:00 +CPU= 36.062
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001116
  22:52:34 STEP 1117 H= 279:15 +CPU= 28.480
  22:53:03 STEP 1118 H= 279:30 +CPU= 28.425
  22:53:32 STEP 1119 H= 279:45 +CPU= 28.900
  22:54:03 STEP 1120 H= 280:00 +CPU= 30.562
  22:54:31 STEP 1121 H= 280:15 +CPU= 28.517
  22:55:00 STEP 1122 H= 280:30 +CPU= 28.361
  22:55:28 STEP 1123 H= 280:45 +CPU= 28.338
  22:55:57 STEP 1124 H= 281:00 +CPU= 28.859
  22:56:26 STEP 1125 H= 281:15 +CPU= 28.568
  22:56:55 STEP 1126 H= 281:30 +CPU= 28.943
  22:57:24 STEP 1127 H= 281:45 +CPU= 28.564
  22:58:01 STEP 1128 H= 282:00 +CPU= 36.389
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001128
  22:58:29 STEP 1129 H= 282:15 +CPU= 28.248
  22:58:59 STEP 1130 H= 282:30 +CPU= 29.201
  22:59:28 STEP 1131 H= 282:45 +CPU= 29.160
  22:59:58 STEP 1132 H= 283:00 +CPU= 29.746
  23:00:28 STEP 1133 H= 283:15 +CPU= 29.961
  23:00:58 STEP 1134 H= 283:30 +CPU= 28.936
  23:01:27 STEP 1135 H= 283:45 +CPU= 28.878
  23:01:57 STEP 1136 H= 284:00 +CPU= 29.850
  23:02:25 STEP 1137 H= 284:15 +CPU= 28.019
  23:02:55 STEP 1138 H= 284:30 +CPU= 28.784
  23:03:25 STEP 1139 H= 284:45 +CPU= 30.162
  23:04:02 STEP 1140 H= 285:00 +CPU= 36.543
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001140
  23:04:32 STEP 1141 H= 285:15 +CPU= 29.525
  23:05:02 STEP 1142 H= 285:30 +CPU= 30.106
  23:05:30 STEP 1143 H= 285:45 +CPU= 28.249
  23:06:00 STEP 1144 H= 286:00 +CPU= 29.514
  23:06:29 STEP 1145 H= 286:15 +CPU= 28.911
  23:06:58 STEP 1146 H= 286:30 +CPU= 28.474
  23:07:28 STEP 1147 H= 286:45 +CPU= 29.707
  23:07:58 STEP 1148 H= 287:00 +CPU= 29.675
  23:08:28 STEP 1149 H= 287:15 +CPU= 29.143
  23:08:57 STEP 1150 H= 287:30 +CPU= 29.097
  23:09:27 STEP 1151 H= 287:45 +CPU= 29.742
  23:10:08 STEP 1152 H= 288:00 +CPU= 40.598
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294554/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1_r1423833186_11.zip
Uploading the intermediate file: upload_file_11.zip
  23:10:40 STEP 1153 H= 288:15 +CPU= 31.164
  23:11:09 STEP 1154 H= 288:30 +CPU= 28.999
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231130:1718493090:38103.390] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' : sigcontextptr=0x7ffee6b08950
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.391] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' : sigcontextptr=0x7ffee6b08860
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.406] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:ubuntu:1:1:3239568:3239568] [20240615:231131:1718493091:38104.406] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.234398488330141E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.354333504306688E-05 0.000000000000000E+00 0.850193317378906E-02
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.262908651152596E-04 0.000000000000000E+00 0.122578261430972E-01
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.234125122981385E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.252846541440154E-04 0.000000000000000E+00 0.256617756543740E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.177075515467531E-04 0.000000000000000E+00 0.642793496776320E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.579023029707175E-05 0.000000000000000E+00 0.205567062165160E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.177782461529425E-05 0.000000000000000E+00 0.115399420929837E-01
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.198955430319888E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1152 STEPO   F00000000
 CREATE RESTART FILE ./srf00120000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00120000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00120000.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    2.7823 SECONDS******** Mbytes/s, TOTAL TIME=    3.4942(79.6%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00111200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1152 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001152 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001152
 MPI-TASK:   1 -    316357972 BYTES IN      6 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0060( 3.3%)
 NSTEP =  1152 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001152 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001152
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0070 SECONDS101057.0 Mbytes/s, TOTAL TIME=    0.2954( 2.4%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001152 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001152
 MPI-TASK:   1 -   5061727552 BYTES IN    125 RECORDS TRANSFERRED IN    0.0048 SECONDS******** Mbytes/s, TOTAL TIME=    0.2243( 2.1%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001152 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001152
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0003 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0030(10.0%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001152 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001152
 MPI-TASK:   1 -    307778243 BYTES IN      4 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0043( 4.7%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1152 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1152 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001152 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001152
 MPI-TASK:   1 -    316357972 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS790894.9 Mbytes/s, TOTAL TIME=    0.0069( 5.8%)
 23:10:08 STEP 1152 H= 288:00 +CPU= 40.598
 NSTEP =  1153 STEPO   0AAA00AAA
 23:10:40 STEP 1153 H= 288:15 +CPU= 31.164
 NSTEP =  1154 STEPO   0AAA00AAA
 23:11:09 STEP 1154 H= 288:30 +CPU= 28.999
ISTEP=    1155 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1155 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1155 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 23:09:33 FULLPOS-B DYNFPOS   1152     1.423    1.423    1.427    547:13    633:06 0.37558479549783E-04       2GB       0MB
 23:09:37 FULLPOS-V DYNFPOS   1152     3.727    3.727    4.000    547:17    633:10 0.37558479549783E-04       2GB       0MB
 23:09:38 FULLPOS-T DYNFPOS   1152     0.787    0.787    0.789    547:17    633:11 0.37558479549783E-04       2GB       0MB
 23:09:38 0AAA00AAA STEPO     1152     0.784    0.784    0.787    547:18    633:11 0.37558479549783E-04       2GB       0MB
 23:10:08 0AAA00AAA STEPO     1153    29.722   29.722   29.839    547:48    633:41 0.37558479549783E-04       2GB       0MB
 23:10:40 0AAA00AAA STEPO     1154    31.164   31.164   31.580    548:19    634:13 0.37558479549783E-04       2GB       0MB
 23:11:09 0AAA00AAA STEPO     1155    28.999   28.999   29.213    548:48    634:42 0.37558479549783E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1152",
 CTIME   = "00120000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1140, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/7/progress_file_12294554.xml

<running_values>
  <last_cpu_time>32944.690000</last_cpu_time>
  <upload_file_number>12</upload_file_number>
  <last_iter>1155</last_iter>
  <last_upload>1036800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Jun 2024 21:32:31 1551095 22449120 oifs_43r3_bl_a4sz_2016092300_20_1018_12294554_1 864,000 27,074 0.0313


©2024 cpdn.org