climateprediction.net home page
Task 22257936

Task 22257936

Name oifs_43r3_ps_0729_1987050100_123_956_12173373_0
Workunit 12173373
Created 21 Dec 2022, 12:02:39 UTC
Sent 2 Jan 2023, 23:25:15 UTC
Report deadline 1 Feb 2023, 23:25:15 UTC
Received 7 Jan 2023, 22:04:18 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1537562
Run time 15 hours 53 min 37 sec
CPU time 15 hours 26 min 27 sec
Validate state Invalid
Credit 12,821.76
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,798.92 MB
Peak swap size 4,980.96 MB
Peak disk usage 7,847.59 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
24
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002124
  19:09:57 STEP 2125 H=2125:00 +CPU= 14.523
  19:10:13 STEP 2126 H=2126:00 +CPU= 14.929
  19:10:38 STEP 2127 H=2127:00 +CPU= 22.663
  19:10:54 STEP 2128 H=2128:00 +CPU= 14.944
  19:11:10 STEP 2129 H=2129:00 +CPU= 14.330
  19:11:35 STEP 2130 H=2130:00 +CPU= 23.225
  19:11:50 STEP 2131 H=2131:00 +CPU= 14.380
  19:12:06 STEP 2132 H=2132:00 +CPU= 14.922
  19:12:30 STEP 2133 H=2133:00 +CPU= 22.982
  19:12:46 STEP 2134 H=2134:00 +CPU= 14.951
  19:13:02 STEP 2135 H=2135:00 +CPU= 14.916
  19:13:31 STEP 2136 H=2136:00 +CPU= 26.671
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_88.zip
Uploading the intermediate file: upload_file_88.zip
  19:13:47 STEP 2137 H=2137:00 +CPU= 14.221
  19:14:03 STEP 2138 H=2138:00 +CPU= 14.784
  19:14:27 STEP 2139 H=2139:00 +CPU= 22.561
  19:14:43 STEP 2140 H=2140:00 +CPU= 14.961
  19:14:59 STEP 2141 H=2141:00 +CPU= 14.457
  19:15:24 STEP 2142 H=2142:00 +CPU= 23.279
  19:15:40 STEP 2143 H=2143:00 +CPU= 14.528
  19:15:56 STEP 2144 H=2144:00 +CPU= 14.985
  19:16:20 STEP 2145 H=2145:00 +CPU= 22.727
  19:16:36 STEP 2146 H=2146:00 +CPU= 14.821
  19:16:51 STEP 2147 H=2147:00 +CPU= 14.463
  19:17:18 STEP 2148 H=2148:00 +CPU= 25.442
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002148
  19:17:34 STEP 2149 H=2149:00 +CPU= 14.422
  19:17:50 STEP 2150 H=2150:00 +CPU= 14.881
  19:18:14 STEP 2151 H=2151:00 +CPU= 22.668
  19:18:30 STEP 2152 H=2152:00 +CPU= 15.000
  19:18:44 STEP 2153 H=2153:00 +CPU= 14.578
  19:19:08 STEP 2154 H=2154:00 +CPU= 23.543
  19:19:23 STEP 2155 H=2155:00 +CPU= 14.598
  19:19:39 STEP 2156 H=2156:00 +CPU= 15.098
  19:20:02 STEP 2157 H=2157:00 +CPU= 23.008
  19:20:17 STEP 2158 H=2158:00 +CPU= 15.061
  19:20:32 STEP 2159 H=2159:00 +CPU= 15.016
  19:20:59 STEP 2160 H=2160:00 +CPU= 26.465
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_89.zip
Uploading the intermediate file: upload_file_89.zip
  19:21:14 STEP 2161 H=2161:00 +CPU= 14.475
  19:21:29 STEP 2162 H=2162:00 +CPU= 14.425
  19:21:52 STEP 2163 H=2163:00 +CPU= 23.054
  19:22:07 STEP 2164 H=2164:00 +CPU= 15.084
  19:22:22 STEP 2165 H=2165:00 +CPU= 14.661
  19:22:47 STEP 2166 H=2166:00 +CPU= 23.664
  19:23:01 STEP 2167 H=2167:00 +CPU= 14.362
  19:23:17 STEP 2168 H=2168:00 +CPU= 15.186
  19:23:40 STEP 2169 H=2169:00 +CPU= 23.045
  19:23:55 STEP 2170 H=2170:00 +CPU= 15.145
  19:24:10 STEP 2171 H=2171:00 +CPU= 14.737
  19:24:37 STEP 2172 H=2172:00 +CPU= 25.904
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002172
  19:24:52 STEP 2173 H=2173:00 +CPU= 14.609
  19:25:08 STEP 2174 H=2174:00 +CPU= 14.981
  19:25:32 STEP 2175 H=2175:00 +CPU= 22.949
  19:25:47 STEP 2176 H=2176:00 +CPU= 15.137
  19:26:02 STEP 2177 H=2177:00 +CPU= 14.585
  19:26:27 STEP 2178 H=2178:00 +CPU= 23.588
  19:26:42 STEP 2179 H=2179:00 +CPU= 14.621
  19:26:57 STEP 2180 H=2180:00 +CPU= 15.040
  19:27:20 STEP 2181 H=2181:00 +CPU= 22.866
  19:27:36 STEP 2182 H=2182:00 +CPU= 15.016
  19:27:51 STEP 2183 H=2183:00 +CPU= 14.905
  19:28:18 STEP 2184 H=2184:00 +CPU= 26.765
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_90.zip
Uploading the intermediate file: upload_file_90.zip
  19:28:33 STEP 2185 H=2185:00 +CPU= 15.164
  19:28:49 STEP 2186 H=2186:00 +CPU= 15.714
  19:29:14 STEP 2187 H=2187:00 +CPU= 24.264
  19:29:30 STEP 2188 H=2188:00 +CPU= 16.326
  19:29:47 STEP 2189 H=2189:00 +CPU= 15.918
  19:30:13 STEP 2190 H=2190:00 +CPU= 25.287
  19:30:29 STEP 2191 H=2191:00 +CPU= 15.920
  19:30:46 STEP 2192 H=2192:00 +CPU= 16.385
  19:31:10 STEP 2193 H=2193:00 +CPU= 24.504
  19:31:27 STEP 2194 H=2194:00 +CPU= 16.102
  19:31:43 STEP 2195 H=2195:00 +CPU= 15.704
  19:32:10 STEP 2196 H=2196:00 +CPU= 27.559
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002196
  19:32:26 STEP 2197 H=2197:00 +CPU= 15.582
  19:32:43 STEP 2198 H=2198:00 +CPU= 16.340
  19:33:08 STEP 2199 H=2199:00 +CPU= 24.482
  19:33:24 STEP 2200 H=2200:00 +CPU= 16.314
  19:33:40 STEP 2201 H=2201:00 +CPU= 15.788
  19:34:07 STEP 2202 H=2202:00 +CPU= 25.773
  19:34:23 STEP 2203 H=2203:00 +CPU= 16.186
  19:34:40 STEP 2204 H=2204:00 +CPU= 17.036
  19:35:06 STEP 2205 H=2205:00 +CPU= 25.768
  19:35:24 STEP 2206 H=2206:00 +CPU= 17.156
  19:35:41 STEP 2207 H=2207:00 +CPU= 16.980
  19:36:12 STEP 2208 H=2208:00 +CPU= 30.313
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_91.zip
Uploading the intermediate file: upload_file_91.zip
  19:36:29 STEP 2209 H=2209:00 +CPU= 16.610
  19:36:46 STEP 2210 H=2210:00 +CPU= 17.372
  19:37:13 STEP 2211 H=2211:00 +CPU= 26.226
  19:37:30 STEP 2212 H=2212:00 +CPU= 17.089
  19:37:47 STEP 2213 H=2213:00 +CPU= 16.625
  19:38:14 STEP 2214 H=2214:00 +CPU= 26.548
  19:38:31 STEP 2215 H=2215:00 +CPU= 16.842
  19:38:49 STEP 2216 H=2216:00 +CPU= 17.002
  19:39:15 STEP 2217 H=2217:00 +CPU= 25.792
  19:39:33 STEP 2218 H=2218:00 +CPU= 17.259
  19:39:50 STEP 2219 H=2219:00 +CPU= 16.336
  19:40:20 STEP 2220 H=2220:00 +CPU= 28.906
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002220
  19:40:37 STEP 2221 H=2221:00 +CPU= 16.721
  19:40:55 STEP 2222 H=2222:00 +CPU= 17.002
  19:41:21 STEP 2223 H=2223:00 +CPU= 25.928
  19:41:39 STEP 2224 H=2224:00 +CPU= 17.090
  19:41:56 STEP 2225 H=2225:00 +CPU= 16.895
  19:42:23 STEP 2226 H=2226:00 +CPU= 26.438
  19:42:40 STEP 2227 H=2227:00 +CPU= 16.772
  19:42:58 STEP 2228 H=2228:00 +CPU= 17.141
  19:43:24 STEP 2229 H=2229:00 +CPU= 25.919
  19:43:42 STEP 2230 H=2230:00 +CPU= 17.111
  19:43:59 STEP 2231 H=2231:00 +CPU= 17.005
  19:44:30 STEP 2232 H=2232:00 +CPU= 30.127
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_92.zip
Uploading the intermediate file: upload_file_92.zip
  19:44:46 STEP 2233 H=2233:00 +CPU= 16.371
  19:45:04 STEP 2234 H=2234:00 +CPU= 17.084
  19:45:30 STEP 2235 H=2235:00 +CPU= 25.528
  19:45:47 STEP 2236 H=2236:00 +CPU= 16.873
  19:46:04 STEP 2237 H=2237:00 +CPU= 16.568
  19:46:30 STEP 2238 H=2238:00 +CPU= 26.468
  19:46:47 STEP 2239 H=2239:00 +CPU= 16.923
  19:47:05 STEP 2240 H=2240:00 +CPU= 17.553
  19:47:32 STEP 2241 H=2241:00 +CPU= 26.225
  19:47:49 STEP 2242 H=2242:00 +CPU= 17.455
  19:48:07 STEP 2243 H=2243:00 +CPU= 16.926
  19:48:36 STEP 2244 H=2244:00 +CPU= 28.953
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002244
  19:48:53 STEP 2245 H=2245:00 +CPU= 16.744
  19:49:10 STEP 2246 H=2246:00 +CPU= 17.099
  19:49:37 STEP 2247 H=2247:00 +CPU= 26.273
  19:49:54 STEP 2248 H=2248:00 +CPU= 17.054
  19:50:11 STEP 2249 H=2249:00 +CPU= 17.003
  19:50:38 STEP 2250 H=2250:00 +CPU= 26.656
  19:50:55 STEP 2251 H=2251:00 +CPU= 16.986
  19:51:13 STEP 2252 H=2252:00 +CPU= 17.018
  19:51:39 STEP 2253 H=2253:00 +CPU= 26.288
  19:51:56 STEP 2254 H=2254:00 +CPU= 17.072
  19:52:14 STEP 2255 H=2255:00 +CPU= 17.199
  19:52:45 STEP 2256 H=2256:00 +CPU= 30.524
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_93.zip
Uploading the intermediate file: upload_file_93.zip
  19:53:02 STEP 2257 H=2257:00 +CPU= 17.003
  19:53:20 STEP 2258 H=2258:00 +CPU= 17.247
  19:53:46 STEP 2259 H=2259:00 +CPU= 26.040
  19:54:04 STEP 2260 H=2260:00 +CPU= 17.641
  19:54:21 STEP 2261 H=2261:00 +CPU= 16.745
  19:54:48 STEP 2262 H=2262:00 +CPU= 26.586
  19:55:04 STEP 2263 H=2263:00 +CPU= 16.614
  19:55:22 STEP 2264 H=2264:00 +CPU= 17.137
  19:55:48 STEP 2265 H=2265:00 +CPU= 26.005
  19:56:05 STEP 2266 H=2266:00 +CPU= 17.198
  19:56:23 STEP 2267 H=2267:00 +CPU= 17.064
  19:56:52 STEP 2268 H=2268:00 +CPU= 29.013
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002268
  19:57:09 STEP 2269 H=2269:00 +CPU= 16.754
  19:57:26 STEP 2270 H=2270:00 +CPU= 17.244
  19:57:53 STEP 2271 H=2271:00 +CPU= 26.152
  19:58:10 STEP 2272 H=2272:00 +CPU= 16.966
  19:58:27 STEP 2273 H=2273:00 +CPU= 16.512
  19:58:54 STEP 2274 H=2274:00 +CPU= 26.592
  19:59:11 STEP 2275 H=2275:00 +CPU= 16.898
  19:59:28 STEP 2276 H=2276:00 +CPU= 17.110
  19:59:55 STEP 2277 H=2277:00 +CPU= 26.360
  20:00:13 STEP 2278 H=2278:00 +CPU= 17.099
  20:00:30 STEP 2279 H=2279:00 +CPU= 17.142
  20:01:02 STEP 2280 H=2280:00 +CPU= 30.462
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_94.zip
Uploading the intermediate file: upload_file_94.zip
  20:01:19 STEP 2281 H=2281:00 +CPU= 16.711
  20:01:37 STEP 2282 H=2282:00 +CPU= 17.284
  20:02:04 STEP 2283 H=2283:00 +CPU= 26.301
  20:02:22 STEP 2284 H=2284:00 +CPU= 17.278
  20:02:39 STEP 2285 H=2285:00 +CPU= 16.841
  20:03:06 STEP 2286 H=2286:00 +CPU= 26.916
  20:03:23 STEP 2287 H=2287:00 +CPU= 16.775
  20:03:41 STEP 2288 H=2288:00 +CPU= 17.543
  20:04:08 STEP 2289 H=2289:00 +CPU= 26.169
  20:04:25 STEP 2290 H=2290:00 +CPU= 17.319
  20:04:43 STEP 2291 H=2291:00 +CPU= 16.906
  20:05:12 STEP 2292 H=2292:00 +CPU= 29.150
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002292
  20:05:29 STEP 2293 H=2293:00 +CPU= 16.849
  20:05:47 STEP 2294 H=2294:00 +CPU= 17.429
  20:06:14 STEP 2295 H=2295:00 +CPU= 26.420
  20:06:32 STEP 2296 H=2296:00 +CPU= 17.474
  20:06:49 STEP 2297 H=2297:00 +CPU= 16.985
  20:07:16 STEP 2298 H=2298:00 +CPU= 26.799
  20:07:34 STEP 2299 H=2299:00 +CPU= 16.718
  20:07:51 STEP 2300 H=2300:00 +CPU= 17.213
  20:08:18 STEP 2301 H=2301:00 +CPU= 26.167
  20:08:36 STEP 2302 H=2302:00 +CPU= 17.339
  20:08:54 STEP 2303 H=2303:00 +CPU= 17.515
  20:09:25 STEP 2304 H=2304:00 +CPU= 30.431
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_95.zip
Uploading the intermediate file: upload_file_95.zip
  20:09:42 STEP 2305 H=2305:00 +CPU= 16.778
Uploading trickle at timestep: 8294400
  20:10:00 STEP 2306 H=2306:00 +CPU= 17.259
  20:10:28 STEP 2307 H=2307:00 +CPU= 27.282
  20:10:46 STEP 2308 H=2308:00 +CPU= 17.791
  20:11:04 STEP 2309 H=2309:00 +CPU= 17.190
  20:11:33 STEP 2310 H=2310:00 +CPU= 27.626
  20:11:51 STEP 2311 H=2311:00 +CPU= 17.298
  20:12:10 STEP 2312 H=2312:00 +CPU= 18.331
  20:12:38 STEP 2313 H=2313:00 +CPU= 27.288
  20:12:57 STEP 2314 H=2314:00 +CPU= 17.948
  20:13:15 STEP 2315 H=2315:00 +CPU= 17.382
  20:13:46 STEP 2316 H=2316:00 +CPU= 30.280
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002316
  20:14:04 STEP 2317 H=2317:00 +CPU= 17.518
  20:14:21 STEP 2318 H=2318:00 +CPU= 17.638
  20:14:49 STEP 2319 H=2319:00 +CPU= 27.000
  20:15:08 STEP 2320 H=2320:00 +CPU= 17.916
  20:15:26 STEP 2321 H=2321:00 +CPU= 18.117
  20:15:56 STEP 2322 H=2322:00 +CPU= 29.204
  20:16:14 STEP 2323 H=2323:00 +CPU= 17.828
  20:16:33 STEP 2324 H=2324:00 +CPU= 18.718
  20:17:02 STEP 2325 H=2325:00 +CPU= 28.475
  20:17:21 STEP 2326 H=2326:00 +CPU= 18.679
  20:17:40 STEP 2327 H=2327:00 +CPU= 18.472
  20:18:14 STEP 2328 H=2328:00 +CPU= 32.832
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_96.zip
Uploading the intermediate file: upload_file_96.zip
  20:18:33 STEP 2329 H=2329:00 +CPU= 18.222
  20:18:52 STEP 2330 H=2330:00 +CPU= 19.164
  20:19:22 STEP 2331 H=2331:00 +CPU= 29.593
  20:19:42 STEP 2332 H=2332:00 +CPU= 19.635
  20:20:02 STEP 2333 H=2333:00 +CPU= 19.036
  20:20:33 STEP 2334 H=2334:00 +CPU= 30.880
  20:20:52 STEP 2335 H=2335:00 +CPU= 18.788
  20:21:12 STEP 2336 H=2336:00 +CPU= 19.805
  20:21:43 STEP 2337 H=2337:00 +CPU= 29.998
  20:22:03 STEP 2338 H=2338:00 +CPU= 19.757
  20:22:22 STEP 2339 H=2339:00 +CPU= 18.887
  20:22:56 STEP 2340 H=2340:00 +CPU= 33.309
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002340
  20:23:16 STEP 2341 H=2341:00 +CPU= 19.265
  20:23:36 STEP 2342 H=2342:00 +CPU= 19.485
  20:24:06 STEP 2343 H=2343:00 +CPU= 30.081
  20:24:26 STEP 2344 H=2344:00 +CPU= 19.781
  20:24:46 STEP 2345 H=2345:00 +CPU= 19.210
  20:25:17 STEP 2346 H=2346:00 +CPU= 30.557
  20:25:36 STEP 2347 H=2347:00 +CPU= 19.150
  20:25:56 STEP 2348 H=2348:00 +CPU= 19.531
  20:26:27 STEP 2349 H=2349:00 +CPU= 29.924
  20:26:47 STEP 2350 H=2350:00 +CPU= 20.080
  20:27:08 STEP 2351 H=2351:00 +CPU= 19.959
  20:27:43 STEP 2352 H=2352:00 +CPU= 34.513
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_97.zip
Uploading the intermediate file: upload_file_97.zip
  20:28:03 STEP 2353 H=2353:00 +CPU= 18.900
  20:28:22 STEP 2354 H=2354:00 +CPU= 19.430
  20:28:53 STEP 2355 H=2355:00 +CPU= 29.637
  20:29:13 STEP 2356 H=2356:00 +CPU= 19.588
  20:29:32 STEP 2357 H=2357:00 +CPU= 19.043
  20:30:04 STEP 2358 H=2358:00 +CPU= 30.515
  20:30:23 STEP 2359 H=2359:00 +CPU= 18.928
  20:30:43 STEP 2360 H=2360:00 +CPU= 19.699
  20:31:14 STEP 2361 H=2361:00 +CPU= 29.944
  20:31:34 STEP 2362 H=2362:00 +CPU= 19.541
  20:31:54 STEP 2363 H=2363:00 +CPU= 19.100
  20:32:28 STEP 2364 H=2364:00 +CPU= 33.375
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002364
  20:32:47 STEP 2365 H=2365:00 +CPU= 18.816
  20:33:08 STEP 2366 H=2366:00 +CPU= 19.681
  20:33:38 STEP 2367 H=2367:00 +CPU= 29.871
  20:33:58 STEP 2368 H=2368:00 +CPU= 19.377
  20:34:17 STEP 2369 H=2369:00 +CPU= 18.801
  20:34:46 STEP 2370 H=2370:00 +CPU= 27.912
  20:35:02 STEP 2371 H=2371:00 +CPU= 15.983
  20:35:19 STEP 2372 H=2372:00 +CPU= 16.668
  20:35:44 STEP 2373 H=2373:00 +CPU= 24.760
  20:36:01 STEP 2374 H=2374:00 +CPU= 16.630
  20:36:18 STEP 2375 H=2375:00 +CPU= 16.434
  20:36:49 STEP 2376 H=2376:00 +CPU= 29.397
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_98.zip
Uploading the intermediate file: upload_file_98.zip
  20:37:05 STEP 2377 H=2377:00 +CPU= 16.007
  20:37:22 STEP 2378 H=2378:00 +CPU= 16.750
  20:37:47 STEP 2379 H=2379:00 +CPU= 24.776
  20:38:05 STEP 2380 H=2380:00 +CPU= 16.846
  20:38:21 STEP 2381 H=2381:00 +CPU= 15.894
  20:38:47 STEP 2382 H=2382:00 +CPU= 25.737
  20:39:04 STEP 2383 H=2383:00 +CPU= 15.906
  20:39:21 STEP 2384 H=2384:00 +CPU= 16.748
  20:39:46 STEP 2385 H=2385:00 +CPU= 24.985
  20:40:03 STEP 2386 H=2386:00 +CPU= 16.615
  20:40:20 STEP 2387 H=2387:00 +CPU= 16.004
  20:40:49 STEP 2388 H=2388:00 +CPU= 28.323
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002388
  20:41:05 STEP 2389 H=2389:00 +CPU= 16.351
  20:41:22 STEP 2390 H=2390:00 +CPU= 16.527
  20:41:48 STEP 2391 H=2391:00 +CPU= 25.474
  20:42:05 STEP 2392 H=2392:00 +CPU= 16.803
  20:42:22 STEP 2393 H=2393:00 +CPU= 16.332
  20:42:48 STEP 2394 H=2394:00 +CPU= 25.910
  20:43:05 STEP 2395 H=2395:00 +CPU= 16.207
  20:43:22 STEP 2396 H=2396:00 +CPU= 16.547
  20:43:48 STEP 2397 H=2397:00 +CPU= 25.358
  20:44:05 STEP 2398 H=2398:00 +CPU= 16.556
  20:44:22 STEP 2399 H=2399:00 +CPU= 16.844
  20:44:52 STEP 2400 H=2400:00 +CPU= 29.332
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_99.zip
Uploading the intermediate file: upload_file_99.zip
  20:45:08 STEP 2401 H=2401:00 +CPU= 15.744
  20:45:25 STEP 2402 H=2402:00 +CPU= 16.697
  20:45:50 STEP 2403 H=2403:00 +CPU= 24.856
  20:46:08 STEP 2404 H=2404:00 +CPU= 16.725
  20:46:24 STEP 2405 H=2405:00 +CPU= 15.950
  20:46:50 STEP 2406 H=2406:00 +CPU= 25.762
  20:47:06 STEP 2407 H=2407:00 +CPU= 15.878
  20:47:24 STEP 2408 H=2408:00 +CPU= 16.852
  20:47:49 STEP 2409 H=2409:00 +CPU= 25.140
  20:48:06 STEP 2410 H=2410:00 +CPU= 16.579
  20:48:23 STEP 2411 H=2411:00 +CPU= 16.335
  20:48:52 STEP 2412 H=2412:00 +CPU= 28.344
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002412
  20:49:09 STEP 2413 H=2413:00 +CPU= 16.437
  20:49:26 STEP 2414 H=2414:00 +CPU= 16.809
  20:49:52 STEP 2415 H=2415:00 +CPU= 25.047
  20:50:09 STEP 2416 H=2416:00 +CPU= 16.717
  20:50:26 STEP 2417 H=2417:00 +CPU= 16.437
  20:50:52 STEP 2418 H=2418:00 +CPU= 26.047
  20:51:09 STEP 2419 H=2419:00 +CPU= 15.795
  20:51:25 STEP 2420 H=2420:00 +CPU= 16.402
  20:51:50 STEP 2421 H=2421:00 +CPU= 24.797
  20:52:07 STEP 2422 H=2422:00 +CPU= 16.217
  20:52:23 STEP 2423 H=2423:00 +CPU= 16.327
  20:52:53 STEP 2424 H=2424:00 +CPU= 29.188
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_100.zip
Uploading the intermediate file: upload_file_100.zip
  20:53:09 STEP 2425 H=2425:00 +CPU= 15.730
  20:53:26 STEP 2426 H=2426:00 +CPU= 16.411
  20:53:51 STEP 2427 H=2427:00 +CPU= 24.353
  20:54:07 STEP 2428 H=2428:00 +CPU= 16.002
  20:54:23 STEP 2429 H=2429:00 +CPU= 15.863
  20:54:48 STEP 2430 H=2430:00 +CPU= 25.154
  20:55:04 STEP 2431 H=2431:00 +CPU= 15.664
  20:55:21 STEP 2432 H=2432:00 +CPU= 16.154
  20:55:45 STEP 2433 H=2433:00 +CPU= 24.291
  20:56:02 STEP 2434 H=2434:00 +CPU= 16.214
  20:56:17 STEP 2435 H=2435:00 +CPU= 15.517
  20:56:45 STEP 2436 H=2436:00 +CPU= 27.444
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002436
  20:57:01 STEP 2437 H=2437:00 +CPU= 15.708
  20:57:17 STEP 2438 H=2438:00 +CPU= 16.139
  20:57:42 STEP 2439 H=2439:00 +CPU= 24.458
  20:57:59 STEP 2440 H=2440:00 +CPU= 16.417
  20:58:15 STEP 2441 H=2441:00 +CPU= 15.670
  20:58:41 STEP 2442 H=2442:00 +CPU= 25.321
  20:58:57 STEP 2443 H=2443:00 +CPU= 15.861
  20:59:13 STEP 2444 H=2444:00 +CPU= 16.234
  20:59:39 STEP 2445 H=2445:00 +CPU= 24.637
  20:59:55 STEP 2446 H=2446:00 +CPU= 16.270
  21:00:12 STEP 2447 H=2447:00 +CPU= 16.495
  21:00:42 STEP 2448 H=2448:00 +CPU= 28.776
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_101.zip
Uploading the intermediate file: upload_file_101.zip
  21:00:58 STEP 2449 H=2449:00 +CPU= 15.994
  21:01:15 STEP 2450 H=2450:00 +CPU= 16.188
  21:01:40 STEP 2451 H=2451:00 +CPU= 24.820
  21:01:57 STEP 2452 H=2452:00 +CPU= 16.443
  21:02:13 STEP 2453 H=2453:00 +CPU= 15.649
  21:02:39 STEP 2454 H=2454:00 +CPU= 25.369
  21:02:55 STEP 2455 H=2455:00 +CPU= 15.643
  21:03:12 STEP 2456 H=2456:00 +CPU= 16.300
  21:03:36 STEP 2457 H=2457:00 +CPU= 24.499
  21:03:53 STEP 2458 H=2458:00 +CPU= 16.379
  21:04:09 STEP 2459 H=2459:00 +CPU= 15.521
  21:04:37 STEP 2460 H=2460:00 +CPU= 27.299
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002460
  21:04:53 STEP 2461 H=2461:00 +CPU= 15.694
  21:05:09 STEP 2462 H=2462:00 +CPU= 16.163
  21:05:34 STEP 2463 H=2463:00 +CPU= 24.256
  21:05:50 STEP 2464 H=2464:00 +CPU= 16.073
  21:06:06 STEP 2465 H=2465:00 +CPU= 15.776
  21:06:31 STEP 2466 H=2466:00 +CPU= 24.915
  21:06:47 STEP 2467 H=2467:00 +CPU= 15.571
  21:07:04 STEP 2468 H=2468:00 +CPU= 16.278
  21:07:29 STEP 2469 H=2469:00 +CPU= 24.512
  21:07:45 STEP 2470 H=2470:00 +CPU= 16.187
  21:08:02 STEP 2471 H=2471:00 +CPU= 16.280
  21:08:31 STEP 2472 H=2472:00 +CPU= 28.667
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_102.zip
Uploading the intermediate file: upload_file_102.zip
  21:08:47 STEP 2473 H=2473:00 +CPU= 15.692
  21:09:03 STEP 2474 H=2474:00 +CPU= 16.059
  21:09:28 STEP 2475 H=2475:00 +CPU= 24.544
  21:09:45 STEP 2476 H=2476:00 +CPU= 16.276
  21:10:00 STEP 2477 H=2477:00 +CPU= 15.673
  21:10:26 STEP 2478 H=2478:00 +CPU= 24.952
  21:10:42 STEP 2479 H=2479:00 +CPU= 15.698
  21:10:58 STEP 2480 H=2480:00 +CPU= 15.853
  21:11:22 STEP 2481 H=2481:00 +CPU= 23.887
  21:11:39 STEP 2482 H=2482:00 +CPU= 15.860
  21:11:55 STEP 2483 H=2483:00 +CPU= 15.392
  21:12:22 STEP 2484 H=2484:00 +CPU= 26.856
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002484
  21:12:38 STEP 2485 H=2485:00 +CPU= 15.451
  21:12:54 STEP 2486 H=2486:00 +CPU= 15.751
  21:13:19 STEP 2487 H=2487:00 +CPU= 24.006
  21:13:35 STEP 2488 H=2488:00 +CPU= 15.839
  21:13:50 STEP 2489 H=2489:00 +CPU= 15.331
  21:14:15 STEP 2490 H=2490:00 +CPU= 24.624
  21:14:31 STEP 2491 H=2491:00 +CPU= 15.349
  21:14:47 STEP 2492 H=2492:00 +CPU= 15.904
  21:15:11 STEP 2493 H=2493:00 +CPU= 23.840
  21:15:28 STEP 2494 H=2494:00 +CPU= 16.070
  21:15:44 STEP 2495 H=2495:00 +CPU= 15.828
  21:16:13 STEP 2496 H=2496:00 +CPU= 27.748
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_103.zip
Uploading the intermediate file: upload_file_103.zip
  21:16:28 STEP 2497 H=2497:00 +CPU= 14.882
  21:16:44 STEP 2498 H=2498:00 +CPU= 15.600
  21:17:08 STEP 2499 H=2499:00 +CPU= 23.457
  21:17:24 STEP 2500 H=2500:00 +CPU= 15.510
  21:17:39 STEP 2501 H=2501:00 +CPU= 15.086
  21:18:04 STEP 2502 H=2502:00 +CPU= 24.221
  21:18:20 STEP 2503 H=2503:00 +CPU= 15.155
  21:20:19 STEP 2504 H=2504:00 +CPU= 15.125
  21:20:42 STEP 2505 H=2505:00 +CPU= 22.673
  21:20:57 STEP 2506 H=2506:00 +CPU= 15.025
  21:21:12 STEP 2507 H=2507:00 +CPU= 14.501
  21:21:38 STEP 2508 H=2508:00 +CPU= 25.533
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002508
  21:22:02 STEP 2509 H=2509:00 +CPU= 14.196
  21:22:17 STEP 2510 H=2510:00 +CPU= 14.912
  21:22:40 STEP 2511 H=2511:00 +CPU= 22.822
  21:22:55 STEP 2512 H=2512:00 +CPU= 14.950
  21:23:10 STEP 2513 H=2513:00 +CPU= 14.281
  21:23:33 STEP 2514 H=2514:00 +CPU= 23.266
  21:23:48 STEP 2515 H=2515:00 +CPU= 14.519
  21:24:03 STEP 2516 H=2516:00 +CPU= 15.068
  21:24:26 STEP 2517 H=2517:00 +CPU= 22.713
  21:24:41 STEP 2518 H=2518:00 +CPU= 14.832
  21:24:57 STEP 2519 H=2519:00 +CPU= 14.875
  21:25:24 STEP 2520 H=2520:00 +CPU= 26.803
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_104.zip
Uploading the intermediate file: upload_file_104.zip
  21:25:38 STEP 2521 H=2521:00 +CPU= 14.274
  21:25:53 STEP 2522 H=2522:00 +CPU= 14.942
  21:26:17 STEP 2523 H=2523:00 +CPU= 22.861
  21:26:32 STEP 2524 H=2524:00 +CPU= 14.820
  21:26:46 STEP 2525 H=2525:00 +CPU= 14.423
  21:27:10 STEP 2526 H=2526:00 +CPU= 23.332
  21:27:24 STEP 2527 H=2527:00 +CPU= 14.377
  21:27:40 STEP 2528 H=2528:00 +CPU= 14.893
  21:28:03 STEP 2529 H=2529:00 +CPU= 22.632
  21:28:18 STEP 2530 H=2530:00 +CPU= 14.944
  21:28:32 STEP 2531 H=2531:00 +CPU= 14.310
  21:28:58 STEP 2532 H=2532:00 +CPU= 25.532
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002532
  21:29:13 STEP 2533 H=2533:00 +CPU= 14.407
  21:29:28 STEP 2534 H=2534:00 +CPU= 14.951
  21:29:51 STEP 2535 H=2535:00 +CPU= 22.701
  21:30:06 STEP 2536 H=2536:00 +CPU= 14.856
  21:30:21 STEP 2537 H=2537:00 +CPU= 14.308
  21:30:44 STEP 2538 H=2538:00 +CPU= 23.193
  21:30:59 STEP 2539 H=2539:00 +CPU= 14.359
  21:31:14 STEP 2540 H=2540:00 +CPU= 14.839
  21:31:37 STEP 2541 H=2541:00 +CPU= 22.614
  21:31:52 STEP 2542 H=2542:00 +CPU= 14.858
  21:32:07 STEP 2543 H=2543:00 +CPU= 14.836
  21:32:34 STEP 2544 H=2544:00 +CPU= 26.644
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_105.zip
Uploading the intermediate file: upload_file_105.zip
  21:32:49 STEP 2545 H=2545:00 +CPU= 14.367
Uploading trickle at timestep: 9158400
  21:33:04 STEP 2546 H=2546:00 +CPU= 14.864
  21:33:27 STEP 2547 H=2547:00 +CPU= 22.777
  21:33:43 STEP 2548 H=2548:00 +CPU= 14.942
  21:33:57 STEP 2549 H=2549:00 +CPU= 14.401
  21:34:21 STEP 2550 H=2550:00 +CPU= 23.272
  21:34:36 STEP 2551 H=2551:00 +CPU= 14.455
  21:34:51 STEP 2552 H=2552:00 +CPU= 14.965
  21:35:14 STEP 2553 H=2553:00 +CPU= 22.781
  21:35:30 STEP 2554 H=2554:00 +CPU= 14.984
  21:35:44 STEP 2555 H=2555:00 +CPU= 14.436
  21:36:11 STEP 2556 H=2556:00 +CPU= 25.595
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002556
  21:36:25 STEP 2557 H=2557:00 +CPU= 14.398
  21:36:40 STEP 2558 H=2558:00 +CPU= 14.949
  21:37:03 STEP 2559 H=2559:00 +CPU= 22.726
  21:37:19 STEP 2560 H=2560:00 +CPU= 14.954
  21:37:33 STEP 2561 H=2561:00 +CPU= 14.388
  21:37:57 STEP 2562 H=2562:00 +CPU= 23.262
  21:38:11 STEP 2563 H=2563:00 +CPU= 14.401
  21:38:26 STEP 2564 H=2564:00 +CPU= 14.923
  21:38:49 STEP 2565 H=2565:00 +CPU= 22.660
  21:39:04 STEP 2566 H=2566:00 +CPU= 14.889
  21:39:19 STEP 2567 H=2567:00 +CPU= 14.868
  21:39:46 STEP 2568 H=2568:00 +CPU= 26.318
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_106.zip
Uploading the intermediate file: upload_file_106.zip
  21:40:00 STEP 2569 H=2569:00 +CPU= 14.368
  21:40:15 STEP 2570 H=2570:00 +CPU= 14.910
  21:40:38 STEP 2571 H=2571:00 +CPU= 22.715
  21:40:53 STEP 2572 H=2572:00 +CPU= 14.911
  21:41:08 STEP 2573 H=2573:00 +CPU= 14.417
  21:41:31 STEP 2574 H=2574:00 +CPU= 23.303
  21:41:46 STEP 2575 H=2575:00 +CPU= 14.404
  21:42:01 STEP 2576 H=2576:00 +CPU= 14.976
  21:42:24 STEP 2577 H=2577:00 +CPU= 22.711
  21:42:39 STEP 2578 H=2578:00 +CPU= 14.758
  21:42:53 STEP 2579 H=2579:00 +CPU= 14.431
  21:43:19 STEP 2580 H=2580:00 +CPU= 25.550
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002580
  21:43:34 STEP 2581 H=2581:00 +CPU= 14.449
  21:43:49 STEP 2582 H=2582:00 +CPU= 14.857
  21:44:12 STEP 2583 H=2583:00 +CPU= 22.701
  21:44:27 STEP 2584 H=2584:00 +CPU= 14.910
  21:44:41 STEP 2585 H=2585:00 +CPU= 14.391
  21:45:05 STEP 2586 H=2586:00 +CPU= 23.323
  21:45:19 STEP 2587 H=2587:00 +CPU= 14.422
  21:45:34 STEP 2588 H=2588:00 +CPU= 14.921
  21:45:57 STEP 2589 H=2589:00 +CPU= 22.721
  21:46:12 STEP 2590 H=2590:00 +CPU= 14.869
  21:46:27 STEP 2591 H=2591:00 +CPU= 14.745
  21:46:54 STEP 2592 H=2592:00 +CPU= 26.625
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_107.zip
Uploading the intermediate file: upload_file_107.zip
  21:47:09 STEP 2593 H=2593:00 +CPU= 14.345
  21:47:24 STEP 2594 H=2594:00 +CPU= 14.876
  21:47:47 STEP 2595 H=2595:00 +CPU= 22.653
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4802MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4802 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLIMAER_LAYER 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214754:1673128074:20277.335] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 RADACA 
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214755:1673128075:20278.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:DarkBox:1:1:9203:9203] [20230107:214755:1673128075:20278.335] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CCB150  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FFE8C9  radaca_                   224  radaca.F90
oifs_43r3_model.e  0000000000FF616B  climaer_layer_            120  climaer_layer.F90
oifs_43r3_model.e  0000000000F09C63  callpar_                  787  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0729
(argv4) batchid: 956
(argv5) wuid: 12173373
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/31
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8c53decfc60e022d2fcbf1f452feade1 to: /var/lib/boinc/slots/31/oifs_43r3_ps_0729_1987050100_123_956_12173373.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/31/oifs_43r3_ps_0729_1987050100_123_956_12173373.zip
ic_ancil_file: ic_ancil_12173373
ifsdata_file: ifsdata_12173373
climate_data_file: clim_data_12173373
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0a3f35f72bc20fb350c997bf591c149e to: /var/lib/boinc/slots/31/ic_ancil_12173373.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/31/ic_ancil_12173373.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_35e30cb8fc7d1cd1740c78b48f494a5e to: /var/lib/boinc/slots/31/ifsdata/ifsdata_12173373.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/31/ifsdata/ifsdata_12173373.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/31/159l_2/clim_data_12173373.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/31/159l_2/clim_data_12173373.zip
Checking for progress XML file: /var/lib/boinc/slots/31/progress_file_12173373.xml
Opened progress file ok : /var/lib/boinc/slots/31/progress_file_12173373.xml
-- Model is restarting --
Adjusting last_iter, 2596, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/31/progress_file_12173373.xml
last_cpu_time: 55002
upload_file_number: 108
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242
The child process has been launched with process id: 3750
Executing the command: /var/lib/boinc/slots/31/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:DarkBox:1:1:3750:3750] [20230107:215107:1673128267:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:51:40 STEP 2592 H=2592:00 +CPU= 24.995
  21:51:54 STEP 2593 H=2593:00 +CPU= 13.989
  21:52:08 STEP 2594 H=2594:00 +CPU= 14.419
  21:52:31 STEP 2595 H=2595:00 +CPU= 22.266
  21:52:45 STEP 2596 H=2596:00 +CPU= 14.724
  21:53:00 STEP 2597 H=2597:00 +CPU= 14.215
  21:53:22 STEP 2598 H=2598:00 +CPU= 22.827
  21:53:37 STEP 2599 H=2599:00 +CPU= 14.132
  21:53:51 STEP 2600 H=2600:00 +CPU= 14.626
  21:54:13 STEP 2601 H=2601:00 +CPU= 22.273
  21:54:28 STEP 2602 H=2602:00 +CPU= 14.743
  21:54:42 STEP 2603 H=2603:00 +CPU= 14.141
  21:55:08 STEP 2604 H=2604:00 +CPU= 25.344
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002604
  21:55:22 STEP 2605 H=2605:00 +CPU= 14.250
  21:55:37 STEP 2606 H=2606:00 +CPU= 14.754
  21:55:59 STEP 2607 H=2607:00 +CPU= 22.429
  21:56:14 STEP 2608 H=2608:00 +CPU= 14.799
  21:56:28 STEP 2609 H=2609:00 +CPU= 14.157
  21:56:51 STEP 2610 H=2610:00 +CPU= 23.083
  21:57:06 STEP 2611 H=2611:00 +CPU= 14.311
  21:57:20 STEP 2612 H=2612:00 +CPU= 14.708
  21:57:43 STEP 2613 H=2613:00 +CPU= 22.268
  21:57:57 STEP 2614 H=2614:00 +CPU= 14.641
  21:58:12 STEP 2615 H=2615:00 +CPU= 14.624
  21:58:38 STEP 2616 H=2616:00 +CPU= 26.176
Moving to projects directory: /var/lib/boinc/slots/31/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/31/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/31/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12173373/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0729_1987050100_123_956_12173373_0_r2071456242_108.zip
Uploading the intermediate file: upload_file_108.zip
  21:58:53 STEP 2617 H=2617:00 +CPU= 14.352
  21:59:07 STEP 2618 H=2618:00 +CPU= 14.732
  21:59:30 STEP 2619 H=2619:00 +CPU= 22.394
  21:59:45 STEP 2620 H=2620:00 +CPU= 14.773
  22:00:00 STEP 2621 H=2621:00 +CPU= 14.325
  22:00:24 STEP 2622 H=2622:00 +CPU= 23.092
  22:00:40 STEP 2623 H=2623:00 +CPU= 14.367
  22:00:55 STEP 2624 H=2624:00 +CPU= 14.838
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:06 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 9,158,400 54,108 0.0059
09 May 2023 16:12:03 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 8,294,400 49,325 0.0059
09 May 2023 16:12:00 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 7,430,400 44,764 0.0060
09 May 2023 16:11:56 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 6,566,400 40,408 0.0062
09 May 2023 16:09:33 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 5,443,200 33,743 0.0062
09 May 2023 16:09:28 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 4,579,200 29,469 0.0064
09 May 2023 16:08:13 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 3,196,800 18,637 0.0058
09 May 2023 16:08:09 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 2,332,800 13,697 0.0059
09 May 2023 16:07:17 1537562 22257936 oifs_43r3_ps_0729_1987050100_123_956_12173373_0 1,123,200 6,765 0.0060


©2024 cpdn.org