climateprediction.net home page
Task 22316841

Task 22316841

Name oifs_43r3_01jb_2019110100_123_993_12215429_0
Workunit 12215429
Created 24 Feb 2023, 18:33:04 UTC
Sent 24 Feb 2023, 18:55:57 UTC
Report deadline 25 Apr 2023, 18:55:57 UTC
Received 25 Feb 2023, 6:49:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1533319
Run time 11 hours 20 min 27 sec
CPU time 11 hours 8 min 30 sec
Validate state Invalid
Credit 6,048.00
Device peak FLOPS 4.58 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,405.22 MB
Peak swap size 4,983.13 MB
Peak disk usage 1,271.85 MB
Stderr 
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
s_43r3_12215429/ICMGGhnw4+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_40.zip
Uploading the intermediate file: upload_file_40.zip
  04:05:19 STEP  985 H= 985:00 +CPU= 25.124
  04:05:45 STEP  986 H= 986:00 +CPU= 25.488
  04:06:24 STEP  987 H= 987:00 +CPU= 38.847
  04:06:50 STEP  988 H= 988:00 +CPU= 26.224
  04:07:17 STEP  989 H= 989:00 +CPU= 26.373
  04:07:56 STEP  990 H= 990:00 +CPU= 38.803
  04:08:20 STEP  991 H= 991:00 +CPU= 23.760
  04:08:41 STEP  992 H= 992:00 +CPU= 21.374
  04:09:17 STEP  993 H= 993:00 +CPU= 35.460
  04:09:37 STEP  994 H= 994:00 +CPU= 19.115
  04:09:56 STEP  995 H= 995:00 +CPU= 18.511
  04:10:33 STEP  996 H= 996:00 +CPU= 37.569
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+000996
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+000996
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+000996
  04:10:58 STEP  997 H= 997:00 +CPU= 24.478
  04:11:23 STEP  998 H= 998:00 +CPU= 25.239
  04:11:59 STEP  999 H= 999:00 +CPU= 35.415
  04:12:21 STEP 1000 H=1000:00 +CPU= 20.033
  04:12:42 STEP 1001 H=1001:00 +CPU= 20.764
  04:13:23 STEP 1002 H=1002:00 +CPU= 40.788
  04:13:49 STEP 1003 H=1003:00 +CPU= 25.930
  04:14:17 STEP 1004 H=1004:00 +CPU= 27.265
  04:14:55 STEP 1005 H=1005:00 +CPU= 38.411
  04:15:22 STEP 1006 H=1006:00 +CPU= 26.568
  04:15:49 STEP 1007 H=1007:00 +CPU= 26.521
  04:16:36 STEP 1008 H=1008:00 +CPU= 38.802
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001008
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001008
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_41.zip
Uploading the intermediate file: upload_file_41.zip
  04:16:59 STEP 1009 H=1009:00 +CPU= 23.347
  04:17:20 STEP 1010 H=1010:00 +CPU= 20.518
  04:17:58 STEP 1011 H=1011:00 +CPU= 38.355
  04:18:21 STEP 1012 H=1012:00 +CPU= 22.755
  04:18:45 STEP 1013 H=1013:00 +CPU= 23.274
  04:19:20 STEP 1014 H=1014:00 +CPU= 34.212
  04:19:45 STEP 1015 H=1015:00 +CPU= 25.148
  04:20:11 STEP 1016 H=1016:00 +CPU= 25.906
  04:20:50 STEP 1017 H=1017:00 +CPU= 38.724
  04:21:15 STEP 1018 H=1018:00 +CPU= 23.611
  04:21:33 STEP 1019 H=1019:00 +CPU= 17.990
  04:22:06 STEP 1020 H=1020:00 +CPU= 32.118
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001020
  04:22:30 STEP 1021 H=1021:00 +CPU= 24.340
  04:22:57 STEP 1022 H=1022:00 +CPU= 26.416
  04:23:38 STEP 1023 H=1023:00 +CPU= 39.231
  04:24:04 STEP 1024 H=1024:00 +CPU= 26.327
  04:24:26 STEP 1025 H=1025:00 +CPU= 22.004
  04:25:05 STEP 1026 H=1026:00 +CPU= 38.796
  04:25:30 STEP 1027 H=1027:00 +CPU= 25.067
  04:25:57 STEP 1028 H=1028:00 +CPU= 26.099
  04:26:36 STEP 1029 H=1029:00 +CPU= 38.451
  04:26:58 STEP 1030 H=1030:00 +CPU= 22.413
  04:27:22 STEP 1031 H=1031:00 +CPU= 23.427
  04:28:13 STEP 1032 H=1032:00 +CPU= 46.060
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_42.zip
Uploading the intermediate file: upload_file_42.zip
  04:28:37 STEP 1033 H=1033:00 +CPU= 23.861
  04:29:01 STEP 1034 H=1034:00 +CPU= 23.506
  04:29:41 STEP 1035 H=1035:00 +CPU= 39.580
  04:30:08 STEP 1036 H=1036:00 +CPU= 26.395
  04:30:27 STEP 1037 H=1037:00 +CPU= 19.158
  04:31:04 STEP 1038 H=1038:00 +CPU= 36.584
  04:31:29 STEP 1039 H=1039:00 +CPU= 24.957
  04:31:52 STEP 1040 H=1040:00 +CPU= 22.405
  04:32:21 STEP 1041 H=1041:00 +CPU= 29.378
  04:32:44 STEP 1042 H=1042:00 +CPU= 22.312
  04:33:09 STEP 1043 H=1043:00 +CPU= 25.456
  04:33:53 STEP 1044 H=1044:00 +CPU= 43.517
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001044
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001044
  04:34:17 STEP 1045 H=1045:00 +CPU= 24.331
  04:34:43 STEP 1046 H=1046:00 +CPU= 25.781
  04:35:22 STEP 1047 H=1047:00 +CPU= 39.428
  04:35:43 STEP 1048 H=1048:00 +CPU= 20.402
  04:36:02 STEP 1049 H=1049:00 +CPU= 18.739
  04:36:42 STEP 1050 H=1050:00 +CPU= 39.559
  04:37:08 STEP 1051 H=1051:00 +CPU= 25.483
  04:37:29 STEP 1052 H=1052:00 +CPU= 21.095
  04:38:03 STEP 1053 H=1053:00 +CPU= 32.831
  04:38:26 STEP 1054 H=1054:00 +CPU= 22.484
  04:38:45 STEP 1055 H=1055:00 +CPU= 19.085
  04:39:27 STEP 1056 H=1056:00 +CPU= 35.298
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001056
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_43.zip
Uploading the intermediate file: upload_file_43.zip
  04:39:53 STEP 1057 H=1057:00 +CPU= 25.450
  04:40:19 STEP 1058 H=1058:00 +CPU= 25.724
  04:40:56 STEP 1059 H=1059:00 +CPU= 37.646
  04:41:22 STEP 1060 H=1060:00 +CPU= 25.423
  04:41:47 STEP 1061 H=1061:00 +CPU= 24.997
  04:42:27 STEP 1062 H=1062:00 +CPU= 39.924
  04:42:53 STEP 1063 H=1063:00 +CPU= 25.816
  04:43:20 STEP 1064 H=1064:00 +CPU= 26.458
  04:43:52 STEP 1065 H=1065:00 +CPU= 32.375
  04:44:18 STEP 1066 H=1066:00 +CPU= 25.812
  04:44:43 STEP 1067 H=1067:00 +CPU= 25.178
  04:45:25 STEP 1068 H=1068:00 +CPU= 41.399
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001068
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001068
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001068
  04:45:44 STEP 1069 H=1069:00 +CPU= 19.003
  04:46:08 STEP 1070 H=1070:00 +CPU= 23.323
  04:46:45 STEP 1071 H=1071:00 +CPU= 36.231
  04:47:11 STEP 1072 H=1072:00 +CPU= 25.993
  04:47:36 STEP 1073 H=1073:00 +CPU= 23.766
  04:48:14 STEP 1074 H=1074:00 +CPU= 37.810
  04:48:40 STEP 1075 H=1075:00 +CPU= 25.776
  04:49:06 STEP 1076 H=1076:00 +CPU= 25.988
  04:49:38 STEP 1077 H=1077:00 +CPU= 31.710
  04:50:01 STEP 1078 H=1078:00 +CPU= 21.844
  04:50:27 STEP 1079 H=1079:00 +CPU= 26.075
  04:51:13 STEP 1080 H=1080:00 +CPU= 40.237
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001080
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001080
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_44.zip
Uploading the intermediate file: upload_file_44.zip
  04:51:39 STEP 1081 H=1081:00 +CPU= 24.810
  04:52:04 STEP 1082 H=1082:00 +CPU= 25.607
  04:52:40 STEP 1083 H=1083:00 +CPU= 34.416
  04:53:00 STEP 1084 H=1084:00 +CPU= 20.125
  04:53:19 STEP 1085 H=1085:00 +CPU= 18.867
  04:53:53 STEP 1086 H=1086:00 +CPU= 33.023
  04:54:19 STEP 1087 H=1087:00 +CPU= 25.696
  04:54:46 STEP 1088 H=1088:00 +CPU= 26.812
  04:55:24 STEP 1089 H=1089:00 +CPU= 38.357
  04:55:50 STEP 1090 H=1090:00 +CPU= 25.114
  04:56:14 STEP 1091 H=1091:00 +CPU= 24.564
  04:56:55 STEP 1092 H=1092:00 +CPU= 41.205
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001092
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001092
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001092
  04:57:21 STEP 1093 H=1093:00 +CPU= 25.436
  04:57:47 STEP 1094 H=1094:00 +CPU= 26.219
  04:58:25 STEP 1095 H=1095:00 +CPU= 37.862
  04:58:50 STEP 1096 H=1096:00 +CPU= 24.344
  04:59:08 STEP 1097 H=1097:00 +CPU= 18.027
  04:59:43 STEP 1098 H=1098:00 +CPU= 33.100
  05:00:06 STEP 1099 H=1099:00 +CPU= 23.298
  05:00:28 STEP 1100 H=1100:00 +CPU= 21.086
  05:01:08 STEP 1101 H=1101:00 +CPU= 39.411
  05:01:34 STEP 1102 H=1102:00 +CPU= 25.756
  05:02:00 STEP 1103 H=1103:00 +CPU= 25.977
  05:02:51 STEP 1104 H=1104:00 +CPU= 45.751
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001104
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001104
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_45.zip
Uploading the intermediate file: upload_file_45.zip
  05:03:17 STEP 1105 H=1105:00 +CPU= 25.515
  05:03:42 STEP 1106 H=1106:00 +CPU= 25.661
  05:04:21 STEP 1107 H=1107:00 +CPU= 38.336
  05:04:46 STEP 1108 H=1108:00 +CPU= 25.559
  05:05:12 STEP 1109 H=1109:00 +CPU= 25.280
  05:05:51 STEP 1110 H=1110:00 +CPU= 38.738
  05:06:12 STEP 1111 H=1111:00 +CPU= 20.787
  05:06:37 STEP 1112 H=1112:00 +CPU= 25.419
  05:07:17 STEP 1113 H=1113:00 +CPU= 39.003
  05:07:42 STEP 1114 H=1114:00 +CPU= 25.432
  05:08:07 STEP 1115 H=1115:00 +CPU= 24.560
  05:08:49 STEP 1116 H=1116:00 +CPU= 41.886
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001116
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001116
  05:09:15 STEP 1117 H=1117:00 +CPU= 26.075
  05:09:35 STEP 1118 H=1118:00 +CPU= 19.721
  05:10:12 STEP 1119 H=1119:00 +CPU= 36.432
  05:10:38 STEP 1120 H=1120:00 +CPU= 25.325
  05:11:03 STEP 1121 H=1121:00 +CPU= 24.644
  05:11:41 STEP 1122 H=1122:00 +CPU= 37.240
  05:12:05 STEP 1123 H=1123:00 +CPU= 24.582
  05:12:31 STEP 1124 H=1124:00 +CPU= 25.155
  05:13:10 STEP 1125 H=1125:00 +CPU= 39.435
  05:13:37 STEP 1126 H=1126:00 +CPU= 26.156
  05:14:03 STEP 1127 H=1127:00 +CPU= 26.323
  05:14:52 STEP 1128 H=1128:00 +CPU= 44.261
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001128
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_46.zip
Uploading the intermediate file: upload_file_46.zip
  05:15:17 STEP 1129 H=1129:00 +CPU= 24.876
  05:15:43 STEP 1130 H=1130:00 +CPU= 25.684
  05:16:22 STEP 1131 H=1131:00 +CPU= 38.495
  05:16:46 STEP 1132 H=1132:00 +CPU= 23.417
  05:17:08 STEP 1133 H=1133:00 +CPU= 21.051
  05:17:49 STEP 1134 H=1134:00 +CPU= 40.773
  05:18:15 STEP 1135 H=1135:00 +CPU= 25.891
  05:18:35 STEP 1136 H=1136:00 +CPU= 19.907
  05:19:15 STEP 1137 H=1137:00 +CPU= 39.468
  05:19:41 STEP 1138 H=1138:00 +CPU= 25.721
  05:20:06 STEP 1139 H=1139:00 +CPU= 24.844
  05:20:49 STEP 1140 H=1140:00 +CPU= 42.588
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001140
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001140
  05:21:13 STEP 1141 H=1141:00 +CPU= 24.278
  05:21:39 STEP 1142 H=1142:00 +CPU= 24.956
  05:22:10 STEP 1143 H=1143:00 +CPU= 31.453
  05:22:39 STEP 1144 H=1144:00 +CPU= 26.110
  05:23:04 STEP 1145 H=1145:00 +CPU= 25.222
  05:23:40 STEP 1146 H=1146:00 +CPU= 35.730
  05:24:05 STEP 1147 H=1147:00 +CPU= 24.586
  05:24:32 STEP 1148 H=1148:00 +CPU= 26.659
  05:25:12 STEP 1149 H=1149:00 +CPU= 39.976
  05:25:33 STEP 1150 H=1150:00 +CPU= 20.113
  05:25:56 STEP 1151 H=1151:00 +CPU= 22.426
  05:26:39 STEP 1152 H=1152:00 +CPU= 41.076
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_47.zip
Uploading the intermediate file: upload_file_47.zip
  05:27:03 STEP 1153 H=1153:00 +CPU= 23.307
  05:27:24 STEP 1154 H=1154:00 +CPU= 21.571
  05:27:58 STEP 1155 H=1155:00 +CPU= 33.909
  05:28:24 STEP 1156 H=1156:00 +CPU= 25.423
  05:28:49 STEP 1157 H=1157:00 +CPU= 25.112
  05:29:31 STEP 1158 H=1158:00 +CPU= 41.552
  05:29:57 STEP 1159 H=1159:00 +CPU= 26.433
  05:30:21 STEP 1160 H=1160:00 +CPU= 22.938
  05:30:53 STEP 1161 H=1161:00 +CPU= 31.580
  05:31:18 STEP 1162 H=1162:00 +CPU= 24.274
  05:31:39 STEP 1163 H=1163:00 +CPU= 20.202
  05:32:21 STEP 1164 H=1164:00 +CPU= 42.599
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001164
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001164
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001164
  05:32:48 STEP 1165 H=1165:00 +CPU= 24.330
  05:33:08 STEP 1166 H=1166:00 +CPU= 19.109
  05:33:41 STEP 1167 H=1167:00 +CPU= 32.629
  05:34:07 STEP 1168 H=1168:00 +CPU= 25.646
  05:34:32 STEP 1169 H=1169:00 +CPU= 25.148
  05:35:10 STEP 1170 H=1170:00 +CPU= 37.468
  05:35:34 STEP 1171 H=1171:00 +CPU= 24.202
  05:36:00 STEP 1172 H=1172:00 +CPU= 25.515
  05:36:33 STEP 1173 H=1173:00 +CPU= 32.900
  05:36:59 STEP 1174 H=1174:00 +CPU= 25.291
  05:37:25 STEP 1175 H=1175:00 +CPU= 26.113
  05:38:16 STEP 1176 H=1176:00 +CPU= 45.372
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001176
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001176
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_48.zip
Uploading the intermediate file: upload_file_48.zip
  05:38:39 STEP 1177 H=1177:00 +CPU= 22.335
  05:39:01 STEP 1178 H=1178:00 +CPU= 22.160
  05:39:40 STEP 1179 H=1179:00 +CPU= 38.186
  05:40:05 STEP 1180 H=1180:00 +CPU= 25.683
  05:40:30 STEP 1181 H=1181:00 +CPU= 24.723
  05:41:11 STEP 1182 H=1182:00 +CPU= 40.608
  05:41:37 STEP 1183 H=1183:00 +CPU= 25.797
  05:42:02 STEP 1184 H=1184:00 +CPU= 24.916
  05:42:36 STEP 1185 H=1185:00 +CPU= 33.460
  05:43:01 STEP 1186 H=1186:00 +CPU= 25.708
  05:43:27 STEP 1187 H=1187:00 +CPU= 25.469
  05:44:09 STEP 1188 H=1188:00 +CPU= 42.276
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001188
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001188
  05:44:35 STEP 1189 H=1189:00 +CPU= 25.367
  05:45:01 STEP 1190 H=1190:00 +CPU= 25.937
  05:45:40 STEP 1191 H=1191:00 +CPU= 39.174
  05:46:04 STEP 1192 H=1192:00 +CPU= 23.489
  05:46:28 STEP 1193 H=1193:00 +CPU= 23.891
  05:47:04 STEP 1194 H=1194:00 +CPU= 36.039
  05:47:26 STEP 1195 H=1195:00 +CPU= 21.355
  05:47:52 STEP 1196 H=1196:00 +CPU= 25.346
  05:48:31 STEP 1197 H=1197:00 +CPU= 38.193
  05:48:56 STEP 1198 H=1198:00 +CPU= 25.840
  05:49:22 STEP 1199 H=1199:00 +CPU= 25.594
  05:50:17 STEP 1200 H=1200:00 +CPU= 40.101
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001200
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_49.zip
Uploading the intermediate file: upload_file_49.zip
  05:50:42 STEP 1201 H=1201:00 +CPU= 24.731
Uploading trickle at timestep: 4320000
  05:51:08 STEP 1202 H=1202:00 +CPU= 26.225
  05:51:47 STEP 1203 H=1203:00 +CPU= 38.265
  05:52:13 STEP 1204 H=1204:00 +CPU= 25.755
  05:52:39 STEP 1205 H=1205:00 +CPU= 25.929
  05:53:10 STEP 1206 H=1206:00 +CPU= 30.921
  05:53:32 STEP 1207 H=1207:00 +CPU= 21.025
  05:53:58 STEP 1208 H=1208:00 +CPU= 26.136
  05:54:30 STEP 1209 H=1209:00 +CPU= 32.077
  05:54:56 STEP 1210 H=1210:00 +CPU= 25.503
  05:55:20 STEP 1211 H=1211:00 +CPU= 24.114
  05:56:00 STEP 1212 H=1212:00 +CPU= 39.772
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001212
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001212
  05:56:19 STEP 1213 H=1213:00 +CPU= 18.797
  05:56:46 STEP 1214 H=1214:00 +CPU= 25.883
  05:57:23 STEP 1215 H=1215:00 +CPU= 37.773
  05:57:46 STEP 1216 H=1216:00 +CPU= 22.686
  05:58:08 STEP 1217 H=1217:00 +CPU= 21.571
  05:58:40 STEP 1218 H=1218:00 +CPU= 32.183
  05:59:05 STEP 1219 H=1219:00 +CPU= 23.546
  05:59:27 STEP 1220 H=1220:00 +CPU= 22.628
  06:00:03 STEP 1221 H=1221:00 +CPU= 35.275
  06:00:29 STEP 1222 H=1222:00 +CPU= 25.638
  06:00:56 STEP 1223 H=1223:00 +CPU= 26.902
  06:01:59 STEP 1224 H=1224:00 +CPU= 39.354
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001224
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_50.zip
Uploading the intermediate file: upload_file_50.zip
  06:02:23 STEP 1225 H=1225:00 +CPU= 23.775
  06:02:49 STEP 1226 H=1226:00 +CPU= 25.871
  06:03:28 STEP 1227 H=1227:00 +CPU= 38.565
  06:03:55 STEP 1228 H=1228:00 +CPU= 26.896
  06:04:20 STEP 1229 H=1229:00 +CPU= 25.223
  06:05:01 STEP 1230 H=1230:00 +CPU= 40.771
  06:05:26 STEP 1231 H=1231:00 +CPU= 24.723
  06:05:52 STEP 1232 H=1232:00 +CPU= 25.389
  06:06:30 STEP 1233 H=1233:00 +CPU= 37.890
  06:06:55 STEP 1234 H=1234:00 +CPU= 25.537
  06:07:20 STEP 1235 H=1235:00 +CPU= 24.532
  06:07:59 STEP 1236 H=1236:00 +CPU= 38.228
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001236
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001236
  06:08:23 STEP 1237 H=1237:00 +CPU= 23.789
  06:08:49 STEP 1238 H=1238:00 +CPU= 24.874
  06:09:21 STEP 1239 H=1239:00 +CPU= 32.128
  06:09:43 STEP 1240 H=1240:00 +CPU= 21.983
  06:10:04 STEP 1241 H=1241:00 +CPU= 20.716
  06:10:45 STEP 1242 H=1242:00 +CPU= 38.988
  06:11:09 STEP 1243 H=1243:00 +CPU= 24.416
  06:11:36 STEP 1244 H=1244:00 +CPU= 26.000
  06:12:15 STEP 1245 H=1245:00 +CPU= 39.061
  06:12:40 STEP 1246 H=1246:00 +CPU= 25.456
  06:13:06 STEP 1247 H=1247:00 +CPU= 25.369
  06:14:03 STEP 1248 H=1248:00 +CPU= 44.282
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001248
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_51.zip
Uploading the intermediate file: upload_file_51.zip
  06:14:28 STEP 1249 H=1249:00 +CPU= 24.511
  06:14:53 STEP 1250 H=1250:00 +CPU= 25.530
  06:15:29 STEP 1251 H=1251:00 +CPU= 35.815
  06:15:55 STEP 1252 H=1252:00 +CPU= 25.999
  06:16:21 STEP 1253 H=1253:00 +CPU= 25.479
  06:17:01 STEP 1254 H=1254:00 +CPU= 39.657
  06:17:26 STEP 1255 H=1255:00 +CPU= 24.792
  06:17:51 STEP 1256 H=1256:00 +CPU= 25.463
  06:18:25 STEP 1257 H=1257:00 +CPU= 33.413
  06:18:46 STEP 1258 H=1258:00 +CPU= 20.167
  06:19:11 STEP 1259 H=1259:00 +CPU= 24.903
  06:19:47 STEP 1260 H=1260:00 +CPU= 35.728
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001260
  06:20:12 STEP 1261 H=1261:00 +CPU= 24.690
  06:20:38 STEP 1262 H=1262:00 +CPU= 25.475
  06:21:06 STEP 1263 H=1263:00 +CPU= 28.302
  06:21:27 STEP 1264 H=1264:00 +CPU= 20.293
  06:21:53 STEP 1265 H=1265:00 +CPU= 25.438
  06:22:33 STEP 1266 H=1266:00 +CPU= 40.043
  06:22:54 STEP 1267 H=1267:00 +CPU= 21.355
  06:23:19 STEP 1268 H=1268:00 +CPU= 24.360
  06:23:59 STEP 1269 H=1269:00 +CPU= 39.684
  06:24:24 STEP 1270 H=1270:00 +CPU= 25.804
  06:24:50 STEP 1271 H=1271:00 +CPU= 25.552
  06:25:39 STEP 1272 H=1272:00 +CPU= 42.724
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_52.zip
Uploading the intermediate file: upload_file_52.zip
  06:26:01 STEP 1273 H=1273:00 +CPU= 22.049
  06:26:27 STEP 1274 H=1274:00 +CPU= 25.880
  06:27:06 STEP 1275 H=1275:00 +CPU= 38.624
  06:27:31 STEP 1276 H=1276:00 +CPU= 25.361
  06:27:55 STEP 1277 H=1277:00 +CPU= 23.891
  06:28:36 STEP 1278 H=1278:00 +CPU= 40.563
  06:29:02 STEP 1279 H=1279:00 +CPU= 25.559
  06:29:29 STEP 1280 H=1280:00 +CPU= 26.832
  06:30:08 STEP 1281 H=1281:00 +CPU= 37.955
  06:30:30 STEP 1282 H=1282:00 +CPU= 22.258
  06:30:56 STEP 1283 H=1283:00 +CPU= 25.131
  06:31:33 STEP 1284 H=1284:00 +CPU= 35.963
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001284
  06:31:58 STEP 1285 H=1285:00 +CPU= 25.269
  06:32:25 STEP 1286 H=1286:00 +CPU= 26.586
  06:33:04 STEP 1287 H=1287:00 +CPU= 38.924
  06:33:26 STEP 1288 H=1288:00 +CPU= 22.235
  06:33:52 STEP 1289 H=1289:00 +CPU= 24.722
  06:34:31 STEP 1290 H=1290:00 +CPU= 39.094
  06:34:57 STEP 1291 H=1291:00 +CPU= 25.839
  06:35:23 STEP 1292 H=1292:00 +CPU= 25.982
  06:35:57 STEP 1293 H=1293:00 +CPU= 33.009
  06:36:17 STEP 1294 H=1294:00 +CPU= 19.527
  06:36:43 STEP 1295 H=1295:00 +CPU= 26.544
  06:37:34 STEP 1296 H=1296:00 +CPU= 44.827
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_53.zip
Uploading the intermediate file: upload_file_53.zip
  06:37:59 STEP 1297 H=1297:00 +CPU= 25.170
  06:38:20 STEP 1298 H=1298:00 +CPU= 20.913
  06:38:58 STEP 1299 H=1299:00 +CPU= 37.490
  06:39:25 STEP 1300 H=1300:00 +CPU= 26.549
  06:39:52 STEP 1301 H=1301:00 +CPU= 26.403
  06:40:33 STEP 1302 H=1302:00 +CPU= 41.057
  06:40:59 STEP 1303 H=1303:00 +CPU= 25.213
  06:41:22 STEP 1304 H=1304:00 +CPU= 21.960
  06:42:02 STEP 1305 H=1305:00 +CPU= 39.960
  06:42:27 STEP 1306 H=1306:00 +CPU= 25.110
  06:42:52 STEP 1307 H=1307:00 +CPU= 24.679
  06:43:34 STEP 1308 H=1308:00 +CPU= 42.601
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001308
  06:44:00 STEP 1309 H=1309:00 +CPU= 25.091
  06:44:26 STEP 1310 H=1310:00 +CPU= 25.691
  06:45:04 STEP 1311 H=1311:00 +CPU= 37.778
  06:45:30 STEP 1312 H=1312:00 +CPU= 25.745
  06:45:56 STEP 1313 H=1313:00 +CPU= 25.424
  06:46:35 STEP 1314 H=1314:00 +CPU= 38.784
  06:47:00 STEP 1315 H=1315:00 +CPU= 24.749
  06:47:27 STEP 1316 H=1316:00 +CPU= 26.954
  06:48:07 STEP 1317 H=1317:00 +CPU= 39.473
  06:48:34 STEP 1318 H=1318:00 +CPU= 26.810
  06:49:01 STEP 1319 H=1319:00 +CPU= 27.445
  06:49:53 STEP 1320 H=1320:00 +CPU= 43.596
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_54.zip
Uploading the intermediate file: upload_file_54.zip
  06:50:18 STEP 1321 H=1321:00 +CPU= 25.073
  06:50:41 STEP 1322 H=1322:00 +CPU= 22.757
  06:51:15 STEP 1323 H=1323:00 +CPU= 33.609
  06:51:41 STEP 1324 H=1324:00 +CPU= 26.245
  06:52:07 STEP 1325 H=1325:00 +CPU= 25.874
  06:52:47 STEP 1326 H=1326:00 +CPU= 39.123
  06:53:12 STEP 1327 H=1327:00 +CPU= 24.986
  06:53:31 STEP 1328 H=1328:00 +CPU= 19.180
  06:54:10 STEP 1329 H=1329:00 +CPU= 38.482
  06:54:37 STEP 1330 H=1330:00 +CPU= 26.476
  06:55:02 STEP 1331 H=1331:00 +CPU= 25.072
  06:55:46 STEP 1332 H=1332:00 +CPU= 43.152
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001332
  06:56:12 STEP 1333 H=1333:00 +CPU= 25.598
  06:56:39 STEP 1334 H=1334:00 +CPU= 27.391
  06:57:12 STEP 1335 H=1335:00 +CPU= 31.954
  06:57:37 STEP 1336 H=1336:00 +CPU= 25.264
  06:58:02 STEP 1337 H=1337:00 +CPU= 24.715
  06:58:42 STEP 1338 H=1338:00 +CPU= 39.179
  06:59:06 STEP 1339 H=1339:00 +CPU= 23.301
  06:59:30 STEP 1340 H=1340:00 +CPU= 24.733
  07:00:11 STEP 1341 H=1341:00 +CPU= 40.607
  07:00:47 STEP 1342 H=1342:00 +CPU= 23.198
  07:01:41 STEP 1343 H=1343:00 +CPU= 23.937
  07:02:45 STEP 1344 H=1344:00 +CPU= 47.655
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_55.zip
Uploading the intermediate file: upload_file_55.zip
  07:03:14 STEP 1345 H=1345:00 +CPU= 27.606
  07:03:44 STEP 1346 H=1346:00 +CPU= 29.670
  07:04:29 STEP 1347 H=1347:00 +CPU= 41.729
  07:04:57 STEP 1348 H=1348:00 +CPU= 28.590
  07:05:25 STEP 1349 H=1349:00 +CPU= 27.291
  07:06:17 STEP 1350 H=1350:00 +CPU= 39.916
  07:06:53 STEP 1351 H=1351:00 +CPU= 27.862
  07:07:21 STEP 1352 H=1352:00 +CPU= 28.502
  07:08:05 STEP 1353 H=1353:00 +CPU= 42.966
  07:08:35 STEP 1354 H=1354:00 +CPU= 29.148
  07:09:03 STEP 1355 H=1355:00 +CPU= 27.584
  07:09:52 STEP 1356 H=1356:00 +CPU= 48.101
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001356
  07:10:21 STEP 1357 H=1357:00 +CPU= 28.663
  07:10:49 STEP 1358 H=1358:00 +CPU= 27.991
  07:11:23 STEP 1359 H=1359:00 +CPU= 33.798
  07:11:50 STEP 1360 H=1360:00 +CPU= 26.801
  07:12:59 STEP 1361 H=1361:00 +CPU= 27.050
  07:13:40 STEP 1362 H=1362:00 +CPU= 39.643
  07:14:08 STEP 1363 H=1363:00 +CPU= 27.035
  07:14:36 STEP 1364 H=1364:00 +CPU= 27.559
  07:15:19 STEP 1365 H=1365:00 +CPU= 43.098
  07:15:47 STEP 1366 H=1366:00 +CPU= 26.969
  07:16:15 STEP 1367 H=1367:00 +CPU= 26.960
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.568327501884839E-06 0.000000000000000E+00 0.254621265392420E-03
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.637584702863995E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM  OZONE               AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.300634011837670E-05 0.268802797205439E-07 0.164562541470189E-04
 GPNORM TENUNOGW             AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.219857383047137E-06 -.244473140100652E-02 0.831641509589126E-03
 GPNORM TENVNOGW             AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.486221228698894E-07 -.553512690508594E-03 0.184366344102260E-02
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.353905279524601E-05 0.000000000000000E+00 0.130620731389441E-02
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.130508029584454E-05 0.000000000000000E+00 0.732055338353557E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.636614396117631E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.237975657110743E-05 0.000000000000000E+00 0.108565355447645E-02
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.214421123150598E-05 0.000000000000000E+00 0.935448910830072E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.487242621690164E-05 0.000000000000000E+00 0.403804278313601E-02
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.157382318666514E-05 0.000000000000000E+00 0.275426913031908E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.137483779844550E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1368 STEPO   F00000000
 CREATE RESTART FILE ./srf00570000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00570000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00570000.0001
 MPI-TASK:   1 - ************ BYTES IN     35 RECORDS TRANSFERRED IN    3.3921 SECONDS******** Mbytes/s, TOTAL TIME=    3.9571(85.7%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00560000.0001
  MAXGPFV : MAX. VALUE =    1.10930296985805     
  MAXGPFV : MAX. VALUE =   0.764144035757934     
 NSTEP =  1368 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+001368 MODE=w
 IO-STREAM CLOSED - ICMGGhnw4+001368
 MPI-TASK:   1 -   1326724255 BYTES IN     27 RECORDS TRANSFERRED IN    0.0093 SECONDS142658.5 Mbytes/s, TOTAL TIME=    0.0978( 9.5%)
 NSTEP =  1368 STEPO   0AA000000
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.072435173414E-01 1.818527364210E-01 5.642331264526E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.288172098800E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  8.89256448020551E-03 -2.27778297364412E+00  2.00846140813896E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  7.19950728591439E-03 -1.00000000000000E+00  1.00000000000000E+00
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHhnw4+001368 MODE=a
 IO-STREAM CLOSED - ./ICMSHhnw4+001368
 MPI-TASK:   1 -   8739913364 BYTES IN    115 RECORDS TRANSFERRED IN    0.0048 SECONDS******** Mbytes/s, TOTAL TIME=    0.2066( 2.3%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAhnw4+001368 MODE=a
 IO-STREAM CLOSED - ./ICMUAhnw4+001368
 MPI-TASK:   1 -    995043189 BYTES IN     19 RECORDS TRANSFERRED IN    0.0008 SECONDS******** Mbytes/s, TOTAL TIME=    0.0244( 3.3%)
 STEPO_FPOS WAS : B00000F00E
 NSTEP =  1368 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.072435173414E-01 1.818527364210E-01 5.642331264526E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.288172098800E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  8.89256448020551E-03 -2.27778297364412E+00  2.00846140813896E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  7.19950728591439E-03 -1.00000000000000E+00  1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 07:14:36 0AAA00AAA STEPO     1365    27.560   27.560   28.064    661:56    678:04 0.11549425388929E-04       2GB       0MB
 07:15:19 0AAA00AAA STEPO     1366    43.098   43.098   43.420    662:39    678:48 0.11549425388929E-04       2GB       0MB
 07:15:47 0AAA00AAA STEPO     1367    26.970   26.970   27.411    663:06    679:15 0.11549425388929E-04       2GB       0MB
 07:16:15 F00000000 STEPO     1368    27.197   27.197   28.542    663:33    679:44 0.11656860847233E-04       2GB       0MB
 07:16:28 0AA000000 STEPO     1368     2.391    2.391   12.516    663:36    679:56 0.11656860847233E-04       2GB       0MB
 07:16:29 FULLPOS-B DYNFPOS   1368     1.297    1.297    1.363    663:37    679:58 0.11656860847233E-04       2GB       0MB
 07:16:34 0AAA00AAA STEPO     1368     1.671    1.671    5.144    663:39    680:03 0.11656860847233E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1368",
 CTIME   = "00570000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98322.9568051082     ,
 GMASSI  =   98322.7936751967     
 /
------------------------------------------------
>>> Printing last 17 lines from file: waminfo

PARAM=10U/10V,                                                          
REPRES=GG,                                                              
ANALYSIS PERIOD =       0                                               
FORECAST PERIOD =    5706000                                            
WIND TIME STEP IN SECONDS =    3600                                     
ANALYSIS FROM 20191101000000 TO 20191101000000                          
ASSIMILATION NO                                                         
NEW FORECAST NO                                                         
STATUS STORM1= UNFINISHED                                               
STATUS STORM2= UNFINISHED                                               
STATUS STORM3= UNFINISHED                                               
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000                  
LAST DATE FOR SPECTRA FILE OUTPUT = 20200303000000                      
BEGIN DATE FOR USING SURFACE CURRENT = 20191101000000                   
PARALLEL RESTART = YES                                                  
NUMBER OF MPI TASKS USED =          1                                   
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/7/progress_file_12215429.xml

<running_values>
  <last_cpu_time>39847.640000</last_cpu_time>
  <upload_file_number>56</upload_file_number>
  <last_iter>1368</last_iter>
  <last_upload>4838400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 01jb
(argv4) batchid: 993
(argv5) wuid: 12215429
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_576994aa522338b00c815f9e9ebd7afc to: /var/lib/boinc-client/slots/7/oifs_43r3_01jb_2019110100_123_993_12215429.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_01jb_2019110100_123_993_12215429.zip
ic_ancil_file: ic_ancil_12215429
ifsdata_file: ifsdata_12215429
climate_data_file: clim_data_12215429
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_65796a50a74007cb6556fd3350f6f71d to: /var/lib/boinc-client/slots/7/ic_ancil_12215429.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12215429.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fabbd0d4d601661911e09cf4d83083d2 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12215429.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12215429.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12215429.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12215429.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12215429.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12215429.xml
-- Model is restarting --
Adjusting last_iter, 1368, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12215429.xml
last_cpu_time: 39847
upload_file_number: 56
last_iter: 1369
last_upload: 4838400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961
The child process has been launched with process id: 20268
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL03:1:1:20268:20268] [20230225:071822:1677305902:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMGGhnw4+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMSHhnw4+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12215429/ICMUAhnw4+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_01jb_2019110100_123_993_12215429_0_r343157961_56.zip
Uploading the intermediate file: upload_file_56.zip
  07:19:14 STEP 1368 H=1368:00 +CPU= 48.755
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhnw4+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhnw4+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhnw4+001368
  07:19:42 STEP 1369 H=1369:00 +CPU= 26.951
  07:20:12 STEP 1370 H=1370:00 +CPU= 29.831
  07:20:56 STEP 1371 H=1371:00 +CPU= 43.806
  07:21:26 STEP 1372 H=1372:00 +CPU= 29.616
  07:21:55 STEP 1373 H=1373:00 +CPU= 29.274
  07:22:41 STEP 1374 H=1374:00 +CPU= 45.885
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01

 NSTEP =  1372 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.156423651637E-01 1.795567248140E-01 5.662599313401E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.407666994365E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.28497402266706E-02 -2.13608787373561E+00  2.52399308411540E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.22210136800044E-02 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228031500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228033000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228034500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228040000  !!!!!!!!!!!! 
 07:21:26 STEP 1372 H=1372:00 +CPU= 29.616
 NSTEP =  1373 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.950901574031E-01 1.838876391458E-01 5.741974226355E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.282245935570E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  6.67540542514476E-03 -2.29513924107998E+00  1.98085033582502E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  5.57217334677677E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228041500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228043000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228044500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228050000  !!!!!!!!!!!! 
 07:21:55 STEP 1373 H=1373:00 +CPU= 29.274
  DATE= 2019   12   28
ISTEP=    1374: Reset RSOLINC= 1361.0304 RIP0= 1407.0907
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.242877686081999E-004
  9.622004323071002E-007  4.997313790396501E-007  9.593326011059999E-010
  2.025453512012000E-009  7.940283000000001E-011
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20191101
       82530  1.076380272952853E-004
SUECSO4: USE AVAILABLE FILES FOR DECADES:   2010 2020
 SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE:   2.290586479600744E-003
 NSTEP =  1374 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.061461989979E-01 1.844187214320E-01 5.748971487607E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.405605080280E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  1.16381143406184E-02 -2.01565639019914E+00  2.21415613079807E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  9.27556042744644E-03 -1.00000000000000E+00  1.00000000000000E+00
  STEPO: evolving YSPSDT_AR1(  1)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228051500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228053000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228054500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20191228060000  !!!!!!!!!!!! 
 07:22:41 STEP 1374 H=1374:00 +CPU= 45.885
ISTEP=    1375 RSOLINC= 1361.0304 RI0= 1366.0000 ZI0= 1361.0304 RIP0= 1407.0947
ISTEP=    1375 ZI0= 1361.0304 ZSEASON= 1.033845150 REA=149597870000.00 RDEASO=147128789807.58
 NSTEP =  1375 STEPO   0AAA00AAA
  SPECTRAL NORMS: Pattern  1
 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 5.142662631316E-01 1.870586080848E-01 5.704668087900E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.520483465749E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  9.13212312108045E-03 -2.15206214001706E+00  2.27573112574431E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  6.63001531002481E-03 -1.00000000000000E+00  1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 07:19:14 0AAA00AAA STEPO     1369    47.757   47.757   48.917      0:51      0:52 0.11656860847233E-04       2GB       0MB
 07:19:42 0AAA00AAA STEPO     1370    26.951   26.951   27.358      1:18      1:19 0.11656860847233E-04       2GB       0MB
 07:20:12 0AAA00AAA STEPO     1371    29.832   29.832   29.936      1:47      1:49 0.11656860847233E-04       2GB       0MB
 07:20:56 0AAA00AAA STEPO     1372    43.805   43.805   44.438      2:31      2:34 0.11656860847233E-04       2GB       0MB
 07:21:26 0AAA00AAA STEPO     1373    29.616   29.616   29.683      3:01      3:03 0.11656860847233E-04       2GB       0MB
 07:21:55 0AAA00AAA STEPO     1374    29.384   29.384   29.458      3:30      3:33 0.11656860847233E-04       2GB       0MB
 07:22:41 0AAA00AAA STEPO     1375    45.776   45.776   46.284      4:16      4:19 0.11656860847233E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1368",
 CTIME   = "00570000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98322.9568051082     ,
 GMASSI  =   98322.7936751967     
 /
------------------------------------------------
>>> Printing last 17 lines from file: waminfo

PARAM=10U/10V,                                                          
REPRES=GG,                                                              
ANALYSIS PERIOD =       0                                               
FORECAST PERIOD =    5706000                                            
WIND TIME STEP IN SECONDS =    3600                                     
ANALYSIS FROM 20191101000000 TO 20191101000000                          
ASSIMILATION NO                                                         
NEW FORECAST NO                                                         
STATUS STORM1= UNFINISHED                                               
STATUS STORM2= UNFINISHED                                               
STATUS STORM3= UNFINISHED                                               
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000                  
LAST DATE FOR SPECTRA FILE OUTPUT = 20200303000000                      
BEGIN DATE FOR USING SURFACE CURRENT = 20191101000000                   
PARALLEL RESTART = YES                                                  
NUMBER OF MPI TASKS USED =          1                                   
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/7/progress_file_12215429.xml

<running_values>
  <last_cpu_time>40110.730000</last_cpu_time>
  <upload_file_number>57</upload_file_number>
  <last_iter>1375</last_iter>
  <last_upload>4928400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:53:16 1533319 22316841 oifs_43r3_01jb_2019110100_123_993_12215429_0 4,320,000 34,885 0.0081
09 May 2023 16:53:06 1533319 22316841 oifs_43r3_01jb_2019110100_123_993_12215429_0 3,456,000 27,981 0.0081
09 May 2023 16:53:03 1533319 22316841 oifs_43r3_01jb_2019110100_123_993_12215429_0 2,592,000 21,013 0.0081
09 May 2023 16:52:57 1533319 22316841 oifs_43r3_01jb_2019110100_123_993_12215429_0 1,728,000 14,065 0.0081
09 May 2023 16:52:49 1533319 22316841 oifs_43r3_01jb_2019110100_123_993_12215429_0 864,000 7,005 0.0081


©2024 climateprediction.net