climateprediction.net home page
Task 22292148

Task 22292148

Name oifs_43r3_ps_0934_2019050100_123_988_12205578_0
Workunit 12205578
Created 25 Dec 2022, 0:36:52 UTC
Sent 5 Jan 2023, 9:53:17 UTC
Report deadline 4 Feb 2023, 9:53:17 UTC
Received 6 Jan 2023, 5:36:12 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532338
Run time 16 hours 28 min 29 sec
CPU time 16 hours 26 min 48 sec
Validate state Invalid
Credit 11,854.08
Device peak FLOPS 5.62 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,802.34 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,623.76 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ing to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_91.zip
Uploading the intermediate file: upload_file_91.zip
  00:34:34 STEP 2209 H=2209:00 +CPU= 15.812
  00:34:50 STEP 2210 H=2210:00 +CPU= 15.867
  00:35:16 STEP 2211 H=2211:00 +CPU= 25.883
  00:35:32 STEP 2212 H=2212:00 +CPU= 16.018
  00:35:48 STEP 2213 H=2213:00 +CPU= 16.107
  00:36:12 STEP 2214 H=2214:00 +CPU= 23.554
  00:36:27 STEP 2215 H=2215:00 +CPU= 15.255
  00:36:42 STEP 2216 H=2216:00 +CPU= 15.026
  00:37:06 STEP 2217 H=2217:00 +CPU= 24.653
  00:37:23 STEP 2218 H=2218:00 +CPU= 16.069
  00:37:38 STEP 2219 H=2219:00 +CPU= 15.324
  00:38:05 STEP 2220 H=2220:00 +CPU= 26.942
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002220
  00:38:20 STEP 2221 H=2221:00 +CPU= 15.010
  00:38:35 STEP 2222 H=2222:00 +CPU= 15.097
  00:38:59 STEP 2223 H=2223:00 +CPU= 24.047
  00:39:14 STEP 2224 H=2224:00 +CPU= 15.483
  00:39:29 STEP 2225 H=2225:00 +CPU= 14.803
  00:39:54 STEP 2226 H=2226:00 +CPU= 24.988
  00:40:09 STEP 2227 H=2227:00 +CPU= 15.110
  00:40:25 STEP 2228 H=2228:00 +CPU= 15.894
  00:40:52 STEP 2229 H=2229:00 +CPU= 26.451
  00:41:10 STEP 2230 H=2230:00 +CPU= 18.160
  00:41:28 STEP 2231 H=2231:00 +CPU= 18.025
  00:41:58 STEP 2232 H=2232:00 +CPU= 30.277
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_92.zip
Uploading the intermediate file: upload_file_92.zip
  00:42:13 STEP 2233 H=2233:00 +CPU= 15.292
  00:42:30 STEP 2234 H=2234:00 +CPU= 16.158
  00:42:56 STEP 2235 H=2235:00 +CPU= 26.684
  00:43:14 STEP 2236 H=2236:00 +CPU= 17.217
  00:43:29 STEP 2237 H=2237:00 +CPU= 15.942
  00:43:56 STEP 2238 H=2238:00 +CPU= 26.188
  00:44:12 STEP 2239 H=2239:00 +CPU= 16.176
  00:44:28 STEP 2240 H=2240:00 +CPU= 16.250
  00:44:54 STEP 2241 H=2241:00 +CPU= 25.744
  00:45:10 STEP 2242 H=2242:00 +CPU= 16.209
  00:45:26 STEP 2243 H=2243:00 +CPU= 15.969
  00:45:56 STEP 2244 H=2244:00 +CPU= 29.911
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002244
  00:46:13 STEP 2245 H=2245:00 +CPU= 17.024
  00:46:30 STEP 2246 H=2246:00 +CPU= 17.127
  00:46:56 STEP 2247 H=2247:00 +CPU= 25.577
  00:47:12 STEP 2248 H=2248:00 +CPU= 16.403
  00:47:30 STEP 2249 H=2249:00 +CPU= 17.444
  00:47:57 STEP 2250 H=2250:00 +CPU= 27.189
  00:48:13 STEP 2251 H=2251:00 +CPU= 16.378
  00:48:31 STEP 2252 H=2252:00 +CPU= 17.685
  00:48:58 STEP 2253 H=2253:00 +CPU= 26.957
  00:49:15 STEP 2254 H=2254:00 +CPU= 17.133
  00:49:32 STEP 2255 H=2255:00 +CPU= 16.782
  00:50:03 STEP 2256 H=2256:00 +CPU= 31.228
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_93.zip
Uploading the intermediate file: upload_file_93.zip
  00:50:18 STEP 2257 H=2257:00 +CPU= 15.235
  00:50:35 STEP 2258 H=2258:00 +CPU= 16.668
  00:51:01 STEP 2259 H=2259:00 +CPU= 26.100
  00:51:18 STEP 2260 H=2260:00 +CPU= 17.006
  00:51:34 STEP 2261 H=2261:00 +CPU= 16.488
  00:52:02 STEP 2262 H=2262:00 +CPU= 27.574
  00:52:18 STEP 2263 H=2263:00 +CPU= 15.676
  00:52:34 STEP 2264 H=2264:00 +CPU= 16.301
  00:52:59 STEP 2265 H=2265:00 +CPU= 25.261
  00:53:15 STEP 2266 H=2266:00 +CPU= 15.657
  00:53:31 STEP 2267 H=2267:00 +CPU= 15.654
  00:53:58 STEP 2268 H=2268:00 +CPU= 27.409
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002268
  00:54:14 STEP 2269 H=2269:00 +CPU= 15.969
  00:54:31 STEP 2270 H=2270:00 +CPU= 17.056
  00:54:57 STEP 2271 H=2271:00 +CPU= 26.181
  00:55:14 STEP 2272 H=2272:00 +CPU= 16.834
  00:55:31 STEP 2273 H=2273:00 +CPU= 16.702
  00:55:58 STEP 2274 H=2274:00 +CPU= 27.487
  00:56:14 STEP 2275 H=2275:00 +CPU= 15.407
  00:56:30 STEP 2276 H=2276:00 +CPU= 16.775
  00:56:56 STEP 2277 H=2277:00 +CPU= 26.065
  00:57:14 STEP 2278 H=2278:00 +CPU= 17.686
  00:57:31 STEP 2279 H=2279:00 +CPU= 17.082
  00:58:01 STEP 2280 H=2280:00 +CPU= 29.943
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_94.zip
Uploading the intermediate file: upload_file_94.zip
  00:58:17 STEP 2281 H=2281:00 +CPU= 15.539
  00:58:33 STEP 2282 H=2282:00 +CPU= 16.238
  00:58:57 STEP 2283 H=2283:00 +CPU= 24.470
  00:59:14 STEP 2284 H=2284:00 +CPU= 16.986
  00:59:31 STEP 2285 H=2285:00 +CPU= 16.795
  00:59:58 STEP 2286 H=2286:00 +CPU= 26.964
  01:00:14 STEP 2287 H=2287:00 +CPU= 16.299
  01:00:31 STEP 2288 H=2288:00 +CPU= 16.789
  01:00:57 STEP 2289 H=2289:00 +CPU= 26.158
  01:01:14 STEP 2290 H=2290:00 +CPU= 16.381
  01:01:31 STEP 2291 H=2291:00 +CPU= 17.096
  01:02:01 STEP 2292 H=2292:00 +CPU= 29.630
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002292
  01:02:16 STEP 2293 H=2293:00 +CPU= 15.884
  01:02:33 STEP 2294 H=2294:00 +CPU= 16.201
  01:02:59 STEP 2295 H=2295:00 +CPU= 26.725
  01:03:17 STEP 2296 H=2296:00 +CPU= 18.056
  01:03:34 STEP 2297 H=2297:00 +CPU= 16.333
  01:04:01 STEP 2298 H=2298:00 +CPU= 26.833
  01:04:16 STEP 2299 H=2299:00 +CPU= 15.748
  01:04:33 STEP 2300 H=2300:00 +CPU= 16.547
  01:04:58 STEP 2301 H=2301:00 +CPU= 25.044
  01:05:14 STEP 2302 H=2302:00 +CPU= 16.125
  01:05:31 STEP 2303 H=2303:00 +CPU= 17.411
  01:06:01 STEP 2304 H=2304:00 +CPU= 29.856
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_95.zip
Uploading the intermediate file: upload_file_95.zip
  01:06:17 STEP 2305 H=2305:00 +CPU= 15.733
  01:06:34 STEP 2306 H=2306:00 +CPU= 16.836
  01:07:00 STEP 2307 H=2307:00 +CPU= 26.200
  01:07:16 STEP 2308 H=2308:00 +CPU= 15.790
  01:07:31 STEP 2309 H=2309:00 +CPU= 15.324
  01:08:00 STEP 2310 H=2310:00 +CPU= 28.296
  01:08:15 STEP 2311 H=2311:00 +CPU= 15.775
  01:08:32 STEP 2312 H=2312:00 +CPU= 16.292
  01:08:58 STEP 2313 H=2313:00 +CPU= 25.959
  01:09:13 STEP 2314 H=2314:00 +CPU= 15.808
  01:09:29 STEP 2315 H=2315:00 +CPU= 15.307
  01:09:58 STEP 2316 H=2316:00 +CPU= 29.733
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002316
  01:10:14 STEP 2317 H=2317:00 +CPU= 15.533
  01:10:30 STEP 2318 H=2318:00 +CPU= 16.068
  01:10:58 STEP 2319 H=2319:00 +CPU= 27.744
  01:11:14 STEP 2320 H=2320:00 +CPU= 16.214
  01:11:34 STEP 2321 H=2321:00 +CPU= 19.631
  01:12:09 STEP 2322 H=2322:00 +CPU= 35.082
  01:12:29 STEP 2323 H=2323:00 +CPU= 20.003
  01:12:50 STEP 2324 H=2324:00 +CPU= 21.029
  01:13:24 STEP 2325 H=2325:00 +CPU= 34.247
  01:13:44 STEP 2326 H=2326:00 +CPU= 19.926
  01:14:02 STEP 2327 H=2327:00 +CPU= 18.037
  01:14:44 STEP 2328 H=2328:00 +CPU= 41.761
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_96.zip
Uploading the intermediate file: upload_file_96.zip
  01:15:07 STEP 2329 H=2329:00 +CPU= 22.964
  01:15:30 STEP 2330 H=2330:00 +CPU= 23.539
  01:16:07 STEP 2331 H=2331:00 +CPU= 36.940
  01:16:32 STEP 2332 H=2332:00 +CPU= 24.554
  01:16:56 STEP 2333 H=2333:00 +CPU= 23.937
  01:17:34 STEP 2334 H=2334:00 +CPU= 37.726
  01:17:57 STEP 2335 H=2335:00 +CPU= 23.082
  01:18:21 STEP 2336 H=2336:00 +CPU= 24.235
  01:18:55 STEP 2337 H=2337:00 +CPU= 33.782
  01:19:19 STEP 2338 H=2338:00 +CPU= 23.911
  01:19:40 STEP 2339 H=2339:00 +CPU= 21.064
  01:20:19 STEP 2340 H=2340:00 +CPU= 39.615
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002340
  01:20:42 STEP 2341 H=2341:00 +CPU= 22.391
  01:21:03 STEP 2342 H=2342:00 +CPU= 21.536
  01:21:38 STEP 2343 H=2343:00 +CPU= 34.350
  01:22:00 STEP 2344 H=2344:00 +CPU= 22.329
  01:22:23 STEP 2345 H=2345:00 +CPU= 22.717
  01:22:58 STEP 2346 H=2346:00 +CPU= 35.277
  01:23:21 STEP 2347 H=2347:00 +CPU= 22.483
  01:23:42 STEP 2348 H=2348:00 +CPU= 21.821
  01:24:18 STEP 2349 H=2349:00 +CPU= 35.135
  01:24:40 STEP 2350 H=2350:00 +CPU= 22.705
  01:25:04 STEP 2351 H=2351:00 +CPU= 23.321
  01:25:48 STEP 2352 H=2352:00 +CPU= 44.015
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_97.zip
Uploading the intermediate file: upload_file_97.zip
  01:26:11 STEP 2353 H=2353:00 +CPU= 23.280
Uploading trickle at timestep: 8467200
  01:26:33 STEP 2354 H=2354:00 +CPU= 22.414
  01:27:06 STEP 2355 H=2355:00 +CPU= 33.099
  01:27:26 STEP 2356 H=2356:00 +CPU= 19.306
  01:27:46 STEP 2357 H=2357:00 +CPU= 19.896
  01:28:16 STEP 2358 H=2358:00 +CPU= 30.679
  01:28:39 STEP 2359 H=2359:00 +CPU= 22.471
  01:29:01 STEP 2360 H=2360:00 +CPU= 22.261
  01:29:36 STEP 2361 H=2361:00 +CPU= 34.936
  01:29:59 STEP 2362 H=2362:00 +CPU= 22.791
  01:30:21 STEP 2363 H=2363:00 +CPU= 21.900
  01:30:59 STEP 2364 H=2364:00 +CPU= 38.018
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002364
  01:31:18 STEP 2365 H=2365:00 +CPU= 19.068
  01:31:40 STEP 2366 H=2366:00 +CPU= 22.280
  01:32:15 STEP 2367 H=2367:00 +CPU= 35.161
  01:32:36 STEP 2368 H=2368:00 +CPU= 20.800
  01:32:58 STEP 2369 H=2369:00 +CPU= 21.963
  01:33:33 STEP 2370 H=2370:00 +CPU= 34.681
  01:33:53 STEP 2371 H=2371:00 +CPU= 20.499
  01:34:16 STEP 2372 H=2372:00 +CPU= 23.051
  01:34:51 STEP 2373 H=2373:00 +CPU= 35.006
  01:35:12 STEP 2374 H=2374:00 +CPU= 20.898
  01:35:32 STEP 2375 H=2375:00 +CPU= 19.908
  01:36:12 STEP 2376 H=2376:00 +CPU= 39.717
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_98.zip
Uploading the intermediate file: upload_file_98.zip
  01:36:32 STEP 2377 H=2377:00 +CPU= 19.850
  01:36:50 STEP 2378 H=2378:00 +CPU= 18.679
  01:37:25 STEP 2379 H=2379:00 +CPU= 34.401
  01:37:48 STEP 2380 H=2380:00 +CPU= 22.973
  01:38:07 STEP 2381 H=2381:00 +CPU= 19.636
  01:38:38 STEP 2382 H=2382:00 +CPU= 30.770
  01:38:57 STEP 2383 H=2383:00 +CPU= 18.549
  01:39:18 STEP 2384 H=2384:00 +CPU= 21.157
  01:39:50 STEP 2385 H=2385:00 +CPU= 32.167
  01:40:09 STEP 2386 H=2386:00 +CPU= 18.915
  01:40:30 STEP 2387 H=2387:00 +CPU= 21.332
  01:41:07 STEP 2388 H=2388:00 +CPU= 36.836
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002388
  01:41:27 STEP 2389 H=2389:00 +CPU= 20.103
  01:41:50 STEP 2390 H=2390:00 +CPU= 22.609
  01:42:23 STEP 2391 H=2391:00 +CPU= 33.490
  01:42:46 STEP 2392 H=2392:00 +CPU= 22.463
  01:43:07 STEP 2393 H=2393:00 +CPU= 21.206
  01:43:38 STEP 2394 H=2394:00 +CPU= 31.311
  01:43:58 STEP 2395 H=2395:00 +CPU= 19.881
  01:44:21 STEP 2396 H=2396:00 +CPU= 22.882
  01:44:56 STEP 2397 H=2397:00 +CPU= 34.952
  01:45:19 STEP 2398 H=2398:00 +CPU= 22.669
  01:45:39 STEP 2399 H=2399:00 +CPU= 20.394
  01:46:20 STEP 2400 H=2400:00 +CPU= 40.766
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_99.zip
Uploading the intermediate file: upload_file_99.zip
  01:46:38 STEP 2401 H=2401:00 +CPU= 18.211
  01:46:58 STEP 2402 H=2402:00 +CPU= 19.297
  01:47:24 STEP 2403 H=2403:00 +CPU= 26.349
  01:47:41 STEP 2404 H=2404:00 +CPU= 16.898
  01:47:57 STEP 2405 H=2405:00 +CPU= 16.179
  01:48:23 STEP 2406 H=2406:00 +CPU= 25.900
  01:48:41 STEP 2407 H=2407:00 +CPU= 17.649
  01:48:58 STEP 2408 H=2408:00 +CPU= 17.599
  01:49:26 STEP 2409 H=2409:00 +CPU= 28.051
  01:49:45 STEP 2410 H=2410:00 +CPU= 18.377
  01:50:02 STEP 2411 H=2411:00 +CPU= 17.294
  01:50:31 STEP 2412 H=2412:00 +CPU= 28.903
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002412
  01:50:46 STEP 2413 H=2413:00 +CPU= 15.488
  01:51:02 STEP 2414 H=2414:00 +CPU= 16.080
  01:51:29 STEP 2415 H=2415:00 +CPU= 27.126
  01:51:47 STEP 2416 H=2416:00 +CPU= 17.295
  01:52:05 STEP 2417 H=2417:00 +CPU= 17.979
  01:52:31 STEP 2418 H=2418:00 +CPU= 26.628
  01:52:47 STEP 2419 H=2419:00 +CPU= 16.024
  01:53:05 STEP 2420 H=2420:00 +CPU= 17.425
  01:53:30 STEP 2421 H=2421:00 +CPU= 25.306
  01:53:47 STEP 2422 H=2422:00 +CPU= 17.279
  01:54:07 STEP 2423 H=2423:00 +CPU= 19.155
  01:54:40 STEP 2424 H=2424:00 +CPU= 33.010
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_100.zip
Uploading the intermediate file: upload_file_100.zip
  01:54:56 STEP 2425 H=2425:00 +CPU= 16.376
  01:55:13 STEP 2426 H=2426:00 +CPU= 17.386
  01:55:41 STEP 2427 H=2427:00 +CPU= 27.133
  01:55:59 STEP 2428 H=2428:00 +CPU= 18.381
  01:56:17 STEP 2429 H=2429:00 +CPU= 17.891
  01:56:45 STEP 2430 H=2430:00 +CPU= 28.055
  01:57:02 STEP 2431 H=2431:00 +CPU= 17.053
  01:57:19 STEP 2432 H=2432:00 +CPU= 16.810
  01:57:45 STEP 2433 H=2433:00 +CPU= 26.090
  01:58:00 STEP 2434 H=2434:00 +CPU= 15.261
  01:58:16 STEP 2435 H=2435:00 +CPU= 16.046
  01:58:45 STEP 2436 H=2436:00 +CPU= 28.499
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002436
  01:59:00 STEP 2437 H=2437:00 +CPU= 15.777
  01:59:16 STEP 2438 H=2438:00 +CPU= 15.891
  01:59:41 STEP 2439 H=2439:00 +CPU= 24.877
  01:59:57 STEP 2440 H=2440:00 +CPU= 15.586
  02:00:12 STEP 2441 H=2441:00 +CPU= 15.118
  02:00:37 STEP 2442 H=2442:00 +CPU= 25.492
  02:00:54 STEP 2443 H=2443:00 +CPU= 16.596
  02:01:11 STEP 2444 H=2444:00 +CPU= 16.564
  02:01:39 STEP 2445 H=2445:00 +CPU= 28.128
  02:01:57 STEP 2446 H=2446:00 +CPU= 17.846
  02:02:15 STEP 2447 H=2447:00 +CPU= 18.225
  02:02:59 STEP 2448 H=2448:00 +CPU= 44.657
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_101.zip
Uploading the intermediate file: upload_file_101.zip
  02:03:22 STEP 2449 H=2449:00 +CPU= 22.407
  02:03:45 STEP 2450 H=2450:00 +CPU= 23.159
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0934
(argv4) batchid: 988
(argv5) wuid: 12205578
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b56f15a07c7dbccd952c873b1e70ae49 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0934_2019050100_123_988_12205578.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0934_2019050100_123_988_12205578.zip
ic_ancil_file: ic_ancil_12205578
ifsdata_file: ifsdata_12205578
climate_data_file: clim_data_12205578
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_48355b529e8580ffeb106379e4f04666 to: /var/lib/boinc-client/slots/20/ic_ancil_12205578.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12205578.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ad41b165fa589433764f0632651f7d31 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205578.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205578.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205578.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205578.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12205578.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12205578.xml
-- Model is restarting --
Adjusting last_iter, 2450, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12205578.xml
last_cpu_time: 52824
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671
The child process has been launched with process id: 588072
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:588072:588072] [20230106:024100:1672972860:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:42:02 STEP 2448 H=2448:00 +CPU= 59.025
  02:42:31 STEP 2449 H=2449:00 +CPU= 28.887
  02:43:00 STEP 2450 H=2450:00 +CPU= 28.946
  02:43:46 STEP 2451 H=2451:00 +CPU= 45.759
  02:44:17 STEP 2452 H=2452:00 +CPU= 31.037
  02:44:49 STEP 2453 H=2453:00 +CPU= 32.365
  02:45:37 STEP 2454 H=2454:00 +CPU= 47.423
  02:46:01 STEP 2455 H=2455:00 +CPU= 24.249
  02:46:27 STEP 2456 H=2456:00 +CPU= 26.257
  02:47:12 STEP 2457 H=2457:00 +CPU= 44.521
  02:47:48 STEP 2458 H=2458:00 +CPU= 36.416
  02:48:23 STEP 2459 H=2459:00 +CPU= 34.238
  02:49:08 STEP 2460 H=2460:00 +CPU= 45.591
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002460
  02:49:34 STEP 2461 H=2461:00 +CPU= 25.900
  02:50:05 STEP 2462 H=2462:00 +CPU= 30.764
  02:50:49 STEP 2463 H=2463:00 +CPU= 43.748
  02:51:16 STEP 2464 H=2464:00 +CPU= 27.359
  02:51:40 STEP 2465 H=2465:00 +CPU= 23.862
  02:52:22 STEP 2466 H=2466:00 +CPU= 41.722
  02:52:52 STEP 2467 H=2467:00 +CPU= 30.061
  02:53:22 STEP 2468 H=2468:00 +CPU= 30.575
  02:54:09 STEP 2469 H=2469:00 +CPU= 46.509
  02:54:40 STEP 2470 H=2470:00 +CPU= 30.914
  02:55:08 STEP 2471 H=2471:00 +CPU= 28.259
  02:55:57 STEP 2472 H=2472:00 +CPU= 48.714
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_102.zip
Uploading the intermediate file: upload_file_102.zip
  02:56:20 STEP 2473 H=2473:00 +CPU= 23.316
  02:56:47 STEP 2474 H=2474:00 +CPU= 26.958
  02:57:31 STEP 2475 H=2475:00 +CPU= 43.762
  02:57:58 STEP 2476 H=2476:00 +CPU= 27.197
  02:58:23 STEP 2477 H=2477:00 +CPU= 24.978
  02:59:08 STEP 2478 H=2478:00 +CPU= 44.613
  02:59:35 STEP 2479 H=2479:00 +CPU= 27.582
  02:59:59 STEP 2480 H=2480:00 +CPU= 23.374
  03:00:36 STEP 2481 H=2481:00 +CPU= 37.478
  03:01:04 STEP 2482 H=2482:00 +CPU= 28.173
  03:01:31 STEP 2483 H=2483:00 +CPU= 26.692
  03:02:14 STEP 2484 H=2484:00 +CPU= 43.401
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002484
  03:02:45 STEP 2485 H=2485:00 +CPU= 30.708
  03:03:17 STEP 2486 H=2486:00 +CPU= 32.056
  03:04:01 STEP 2487 H=2487:00 +CPU= 43.281
  03:04:38 STEP 2488 H=2488:00 +CPU= 36.775
  03:05:07 STEP 2489 H=2489:00 +CPU= 29.184
  03:05:51 STEP 2490 H=2490:00 +CPU= 43.623
  03:06:22 STEP 2491 H=2491:00 +CPU= 31.354
  03:06:57 STEP 2492 H=2492:00 +CPU= 34.222
  03:07:47 STEP 2493 H=2493:00 +CPU= 49.755
  03:08:20 STEP 2494 H=2494:00 +CPU= 33.635
  03:09:09 STEP 2495 H=2495:00 +CPU= 48.179
  03:10:14 STEP 2496 H=2496:00 +CPU= 65.355
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_103.zip
Uploading the intermediate file: upload_file_103.zip
  03:10:42 STEP 2497 H=2497:00 +CPU= 27.797
  03:11:14 STEP 2498 H=2498:00 +CPU= 31.769
  03:11:58 STEP 2499 H=2499:00 +CPU= 43.335
  03:12:24 STEP 2500 H=2500:00 +CPU= 26.637
  03:12:54 STEP 2501 H=2501:00 +CPU= 29.632
  03:13:42 STEP 2502 H=2502:00 +CPU= 47.267
  03:14:16 STEP 2503 H=2503:00 +CPU= 34.078
  03:14:46 STEP 2504 H=2504:00 +CPU= 30.076
  03:15:29 STEP 2505 H=2505:00 +CPU= 42.913
  03:15:54 STEP 2506 H=2506:00 +CPU= 24.527
  03:16:17 STEP 2507 H=2507:00 +CPU= 22.944
  03:17:00 STEP 2508 H=2508:00 +CPU= 43.103
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002508
  03:17:26 STEP 2509 H=2509:00 +CPU= 26.471
  03:17:54 STEP 2510 H=2510:00 +CPU= 28.113
  03:18:33 STEP 2511 H=2511:00 +CPU= 38.767
  03:18:55 STEP 2512 H=2512:00 +CPU= 22.361
  03:19:18 STEP 2513 H=2513:00 +CPU= 22.140
  03:19:56 STEP 2514 H=2514:00 +CPU= 37.881
  03:20:19 STEP 2515 H=2515:00 +CPU= 23.109
  03:20:44 STEP 2516 H=2516:00 +CPU= 25.440
  03:21:22 STEP 2517 H=2517:00 +CPU= 38.399
  03:21:47 STEP 2518 H=2518:00 +CPU= 24.434
  03:22:13 STEP 2519 H=2519:00 +CPU= 26.068
  03:23:03 STEP 2520 H=2520:00 +CPU= 49.775
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_104.zip
Uploading the intermediate file: upload_file_104.zip
  03:23:26 STEP 2521 H=2521:00 +CPU= 23.429
  03:23:50 STEP 2522 H=2522:00 +CPU= 23.057
  03:24:27 STEP 2523 H=2523:00 +CPU= 37.629
  03:24:52 STEP 2524 H=2524:00 +CPU= 24.188
  03:25:15 STEP 2525 H=2525:00 +CPU= 22.883
  03:25:52 STEP 2526 H=2526:00 +CPU= 37.578
  03:26:15 STEP 2527 H=2527:00 +CPU= 22.938
  03:26:40 STEP 2528 H=2528:00 +CPU= 24.537
  03:27:18 STEP 2529 H=2529:00 +CPU= 38.046
  03:27:42 STEP 2530 H=2530:00 +CPU= 24.505
  03:28:06 STEP 2531 H=2531:00 +CPU= 23.317
  03:28:46 STEP 2532 H=2532:00 +CPU= 40.216
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002532
  03:29:09 STEP 2533 H=2533:00 +CPU= 22.856
  03:29:33 STEP 2534 H=2534:00 +CPU= 24.239
  03:30:11 STEP 2535 H=2535:00 +CPU= 37.477
  03:30:37 STEP 2536 H=2536:00 +CPU= 26.691
  03:31:01 STEP 2537 H=2537:00 +CPU= 23.987
  03:31:41 STEP 2538 H=2538:00 +CPU= 39.343
  03:32:04 STEP 2539 H=2539:00 +CPU= 23.565
  03:32:28 STEP 2540 H=2540:00 +CPU= 23.900
  03:33:04 STEP 2541 H=2541:00 +CPU= 36.188
  03:33:28 STEP 2542 H=2542:00 +CPU= 23.626
  03:33:52 STEP 2543 H=2543:00 +CPU= 24.313
  03:34:36 STEP 2544 H=2544:00 +CPU= 43.551
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_105.zip
Uploading the intermediate file: upload_file_105.zip
  03:34:59 STEP 2545 H=2545:00 +CPU= 22.615
  03:35:21 STEP 2546 H=2546:00 +CPU= 21.968
  03:35:57 STEP 2547 H=2547:00 +CPU= 36.157
  03:36:20 STEP 2548 H=2548:00 +CPU= 23.065
  03:36:43 STEP 2549 H=2549:00 +CPU= 22.238
  03:37:19 STEP 2550 H=2550:00 +CPU= 36.703
  03:37:41 STEP 2551 H=2551:00 +CPU= 21.996
  03:38:05 STEP 2552 H=2552:00 +CPU= 23.352
  03:38:41 STEP 2553 H=2553:00 +CPU= 35.796
  03:39:05 STEP 2554 H=2554:00 +CPU= 24.503
  03:39:30 STEP 2555 H=2555:00 +CPU= 24.362
  03:40:14 STEP 2556 H=2556:00 +CPU= 44.562
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002556
  03:40:38 STEP 2557 H=2557:00 +CPU= 24.296
  03:41:01 STEP 2558 H=2558:00 +CPU= 22.749
  03:41:40 STEP 2559 H=2559:00 +CPU= 38.648
  03:42:04 STEP 2560 H=2560:00 +CPU= 23.629
  03:42:25 STEP 2561 H=2561:00 +CPU= 21.409
  03:42:58 STEP 2562 H=2562:00 +CPU= 33.107
  03:43:18 STEP 2563 H=2563:00 +CPU= 20.238
  03:43:40 STEP 2564 H=2564:00 +CPU= 21.619
  03:44:16 STEP 2565 H=2565:00 +CPU= 36.278
  03:44:40 STEP 2566 H=2566:00 +CPU= 23.249
  03:45:00 STEP 2567 H=2567:00 +CPU= 20.474
  03:45:43 STEP 2568 H=2568:00 +CPU= 42.924
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_106.zip
Uploading the intermediate file: upload_file_106.zip
  03:46:04 STEP 2569 H=2569:00 +CPU= 21.278
  03:46:27 STEP 2570 H=2570:00 +CPU= 23.113
  03:47:03 STEP 2571 H=2571:00 +CPU= 36.097
  03:47:26 STEP 2572 H=2572:00 +CPU= 22.421
  03:47:49 STEP 2573 H=2573:00 +CPU= 23.405
  03:48:21 STEP 2574 H=2574:00 +CPU= 31.513
  03:48:40 STEP 2575 H=2575:00 +CPU= 19.314
  03:49:00 STEP 2576 H=2576:00 +CPU= 19.819
  03:49:37 STEP 2577 H=2577:00 +CPU= 36.998
  03:49:58 STEP 2578 H=2578:00 +CPU= 20.601
  03:50:21 STEP 2579 H=2579:00 +CPU= 22.931
  03:50:55 STEP 2580 H=2580:00 +CPU= 34.792
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002580
  03:51:17 STEP 2581 H=2581:00 +CPU= 21.462
  03:51:39 STEP 2582 H=2582:00 +CPU= 21.940
  03:52:13 STEP 2583 H=2583:00 +CPU= 34.152
  03:52:33 STEP 2584 H=2584:00 +CPU= 20.518
  03:52:51 STEP 2585 H=2585:00 +CPU= 17.700
  03:53:31 STEP 2586 H=2586:00 +CPU= 40.123
  03:53:55 STEP 2587 H=2587:00 +CPU= 23.792
  03:54:13 STEP 2588 H=2588:00 +CPU= 17.995
  03:54:44 STEP 2589 H=2589:00 +CPU= 30.926
  03:55:04 STEP 2590 H=2590:00 +CPU= 19.995
  03:55:24 STEP 2591 H=2591:00 +CPU= 20.135
  03:55:59 STEP 2592 H=2592:00 +CPU= 35.078
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_107.zip
Uploading the intermediate file: upload_file_107.zip
  03:56:19 STEP 2593 H=2593:00 +CPU= 19.641
  03:56:40 STEP 2594 H=2594:00 +CPU= 21.366
  03:57:09 STEP 2595 H=2595:00 +CPU= 29.160
  03:57:26 STEP 2596 H=2596:00 +CPU= 16.542
  03:57:42 STEP 2597 H=2597:00 +CPU= 15.650
  03:58:13 STEP 2598 H=2598:00 +CPU= 31.555
  03:58:30 STEP 2599 H=2599:00 +CPU= 17.295
  03:58:50 STEP 2600 H=2600:00 +CPU= 19.738
  03:59:17 STEP 2601 H=2601:00 +CPU= 27.184
  03:59:33 STEP 2602 H=2602:00 +CPU= 15.952
  03:59:50 STEP 2603 H=2603:00 +CPU= 16.715
  04:00:24 STEP 2604 H=2604:00 +CPU= 33.705
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002604
  04:00:43 STEP 2605 H=2605:00 +CPU= 18.826
  04:01:01 STEP 2606 H=2606:00 +CPU= 18.706
  04:01:29 STEP 2607 H=2607:00 +CPU= 28.163
  04:01:46 STEP 2608 H=2608:00 +CPU= 16.338
  04:02:02 STEP 2609 H=2609:00 +CPU= 16.172
  04:02:29 STEP 2610 H=2610:00 +CPU= 26.618
  04:02:45 STEP 2611 H=2611:00 +CPU= 15.891
  04:03:01 STEP 2612 H=2612:00 +CPU= 16.802
  04:03:29 STEP 2613 H=2613:00 +CPU= 27.314
  04:03:46 STEP 2614 H=2614:00 +CPU= 17.164
  04:04:03 STEP 2615 H=2615:00 +CPU= 17.393
  04:04:36 STEP 2616 H=2616:00 +CPU= 33.262
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_108.zip
Uploading the intermediate file: upload_file_108.zip
  04:04:53 STEP 2617 H=2617:00 +CPU= 16.138
  04:05:09 STEP 2618 H=2618:00 +CPU= 16.808
  04:05:36 STEP 2619 H=2619:00 +CPU= 26.276
  04:05:58 STEP 2620 H=2620:00 +CPU= 22.067
  04:06:21 STEP 2621 H=2621:00 +CPU= 23.124
  04:06:55 STEP 2622 H=2622:00 +CPU= 33.672
  04:07:16 STEP 2623 H=2623:00 +CPU= 21.191
  04:07:38 STEP 2624 H=2624:00 +CPU= 22.670
  04:08:13 STEP 2625 H=2625:00 +CPU= 34.722
  04:08:36 STEP 2626 H=2626:00 +CPU= 22.465
  04:08:57 STEP 2627 H=2627:00 +CPU= 21.373
  04:09:35 STEP 2628 H=2628:00 +CPU= 37.470
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002628
  04:09:55 STEP 2629 H=2629:00 +CPU= 20.525
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0934
(argv4) batchid: 988
(argv5) wuid: 12205578
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b56f15a07c7dbccd952c873b1e70ae49 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0934_2019050100_123_988_12205578.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0934_2019050100_123_988_12205578.zip
ic_ancil_file: ic_ancil_12205578
ifsdata_file: ifsdata_12205578
climate_data_file: clim_data_12205578
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_48355b529e8580ffeb106379e4f04666 to: /var/lib/boinc-client/slots/20/ic_ancil_12205578.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12205578.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ad41b165fa589433764f0632651f7d31 to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205578.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205578.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205578.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205578.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12205578.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12205578.xml
-- Model is restarting --
Adjusting last_iter, 2630, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12205578.xml
last_cpu_time: 58156
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671
The child process has been launched with process id: 590751
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:590751:590751] [20230106:041804:1672978684:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:19:02 STEP 2616 H=2616:00 +CPU= 55.334
  04:19:27 STEP 2617 H=2617:00 +CPU= 24.760
  04:19:50 STEP 2618 H=2618:00 +CPU= 23.168
  04:20:29 STEP 2619 H=2619:00 +CPU= 38.531
  04:20:56 STEP 2620 H=2620:00 +CPU= 27.016
  04:21:26 STEP 2621 H=2621:00 +CPU= 30.126
  04:22:21 STEP 2622 H=2622:00 +CPU= 55.445
  04:22:50 STEP 2623 H=2623:00 +CPU= 27.969
  04:23:21 STEP 2624 H=2624:00 +CPU= 31.624
  04:24:06 STEP 2625 H=2625:00 +CPU= 44.317
  04:24:31 STEP 2626 H=2626:00 +CPU= 24.868
  04:24:55 STEP 2627 H=2627:00 +CPU= 24.375
  04:25:42 STEP 2628 H=2628:00 +CPU= 47.148
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002628
  04:26:09 STEP 2629 H=2629:00 +CPU= 26.798
  04:26:36 STEP 2630 H=2630:00 +CPU= 26.879
  04:27:19 STEP 2631 H=2631:00 +CPU= 42.567
  04:27:47 STEP 2632 H=2632:00 +CPU= 28.868
  04:28:15 STEP 2633 H=2633:00 +CPU= 27.701
  04:28:58 STEP 2634 H=2634:00 +CPU= 42.363
  04:29:21 STEP 2635 H=2635:00 +CPU= 23.879
  04:29:48 STEP 2636 H=2636:00 +CPU= 26.153
  04:30:32 STEP 2637 H=2637:00 +CPU= 44.245
  04:31:00 STEP 2638 H=2638:00 +CPU= 28.502
  04:31:37 STEP 2639 H=2639:00 +CPU= 36.179
  04:32:27 STEP 2640 H=2640:00 +CPU= 50.400
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205578/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0934_2019050100_123_988_12205578_0_r1475154671_109.zip
Uploading the intermediate file: upload_file_109.zip
  04:32:54 STEP 2641 H=2641:00 +CPU= 27.046
  04:33:21 STEP 2642 H=2642:00 +CPU= 27.110
  04:34:06 STEP 2643 H=2643:00 +CPU= 44.778
  04:34:33 STEP 2644 H=2644:00 +CPU= 27.113
  04:35:00 STEP 2645 H=2645:00 +CPU= 27.017
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:46 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 8,467,200 50,557 0.0060
09 May 2023 16:09:43 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 7,603,200 45,592 0.0060
09 May 2023 16:09:36 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 6,739,200 40,459 0.0060
09 May 2023 16:09:36 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 6,739,200 40,459 0.0060
09 May 2023 16:09:11 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 4,320,000 21,767 0.0050
09 May 2023 16:09:07 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 3,456,000 17,725 0.0051
09 May 2023 16:09:07 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 3,456,000 17,725 0.0051
09 May 2023 16:09:04 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 2,592,000 12,962 0.0050
09 May 2023 16:09:00 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 1,728,000 8,610 0.0050
09 May 2023 16:09:00 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 1,728,000 8,610 0.0050
09 May 2023 16:08:57 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 864,000 4,519 0.0052
09 May 2023 16:08:57 1532338 22292148 oifs_43r3_ps_0934_2019050100_123_988_12205578_0 864,000 4,519 0.0052


©2024 cpdn.org