climateprediction.net home page
Task 22303619

Task 22303619

Name oifs_43r3_ps_0639_1993050100_123_962_12179283_1
Workunit 12179283
Created 25 Jan 2023, 18:45:59 UTC
Sent 25 Jan 2023, 18:48:15 UTC
Report deadline 26 Mar 2023, 18:48:15 UTC
Received 28 Jan 2023, 12:21:55 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1509499
Run time 10 hours 47 min 31 sec
CPU time 10 hours 32 min 41 sec
Validate state Invalid
Credit 7,378.56
Device peak FLOPS 5.35 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.34 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.28 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
240:00 +CPU= 18.501
  20:04:10 STEP 1241 H=1241:00 +CPU= 17.842
  20:04:39 STEP 1242 H=1242:00 +CPU= 29.025
  20:04:57 STEP 1243 H=1243:00 +CPU= 17.932
  20:05:15 STEP 1244 H=1244:00 +CPU= 18.352
  20:05:44 STEP 1245 H=1245:00 +CPU= 28.425
  20:06:03 STEP 1246 H=1246:00 +CPU= 18.515
  20:06:22 STEP 1247 H=1247:00 +CPU= 18.548
  20:06:55 STEP 1248 H=1248:00 +CPU= 33.081
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_51.zip
Uploading the intermediate file: upload_file_51.zip
  20:07:13 STEP 1249 H=1249:00 +CPU= 17.953
  20:07:32 STEP 1250 H=1250:00 +CPU= 18.545
  20:08:01 STEP 1251 H=1251:00 +CPU= 28.466
  20:08:19 STEP 1252 H=1252:00 +CPU= 18.405
  20:08:37 STEP 1253 H=1253:00 +CPU= 17.975
  20:09:07 STEP 1254 H=1254:00 +CPU= 29.122
  20:09:25 STEP 1255 H=1255:00 +CPU= 18.040
  20:09:43 STEP 1256 H=1256:00 +CPU= 18.528
  20:10:12 STEP 1257 H=1257:00 +CPU= 28.448
  20:10:31 STEP 1258 H=1258:00 +CPU= 18.413
  20:10:49 STEP 1259 H=1259:00 +CPU= 17.794
  20:11:20 STEP 1260 H=1260:00 +CPU= 31.311
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001260
  20:11:39 STEP 1261 H=1261:00 +CPU= 17.941
  20:11:57 STEP 1262 H=1262:00 +CPU= 18.449
  20:12:26 STEP 1263 H=1263:00 +CPU= 28.393
  20:12:45 STEP 1264 H=1264:00 +CPU= 18.485
  20:13:03 STEP 1265 H=1265:00 +CPU= 18.012
  20:13:32 STEP 1266 H=1266:00 +CPU= 28.992
  20:13:50 STEP 1267 H=1267:00 +CPU= 18.002
  20:14:09 STEP 1268 H=1268:00 +CPU= 18.505
  20:14:38 STEP 1269 H=1269:00 +CPU= 28.596
  20:14:57 STEP 1270 H=1270:00 +CPU= 18.560
  20:15:15 STEP 1271 H=1271:00 +CPU= 18.395
  20:15:49 STEP 1272 H=1272:00 +CPU= 33.492
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_52.zip
Uploading the intermediate file: upload_file_52.zip
  20:16:07 STEP 1273 H=1273:00 +CPU= 17.984
  20:16:26 STEP 1274 H=1274:00 +CPU= 18.375
  20:16:54 STEP 1275 H=1275:00 +CPU= 28.403
  20:17:13 STEP 1276 H=1276:00 +CPU= 18.485
  20:17:31 STEP 1277 H=1277:00 +CPU= 17.791
  20:18:00 STEP 1278 H=1278:00 +CPU= 29.084
  20:18:18 STEP 1279 H=1279:00 +CPU= 17.891
  20:18:37 STEP 1280 H=1280:00 +CPU= 18.511
  20:19:06 STEP 1281 H=1281:00 +CPU= 28.358
  20:19:24 STEP 1282 H=1282:00 +CPU= 18.508
  20:19:42 STEP 1283 H=1283:00 +CPU= 17.916
  20:20:14 STEP 1284 H=1284:00 +CPU= 31.513
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001284
  20:20:32 STEP 1285 H=1285:00 +CPU= 17.849
  20:20:51 STEP 1286 H=1286:00 +CPU= 18.581
  20:21:20 STEP 1287 H=1287:00 +CPU= 28.434
  20:21:39 STEP 1288 H=1288:00 +CPU= 18.667
  20:21:57 STEP 1289 H=1289:00 +CPU= 18.159
  20:22:26 STEP 1290 H=1290:00 +CPU= 29.055
  20:22:44 STEP 1291 H=1291:00 +CPU= 17.932
  20:23:03 STEP 1292 H=1292:00 +CPU= 18.403
  20:23:31 STEP 1293 H=1293:00 +CPU= 28.384
  20:23:50 STEP 1294 H=1294:00 +CPU= 18.443
  20:24:09 STEP 1295 H=1295:00 +CPU= 18.350
  20:24:42 STEP 1296 H=1296:00 +CPU= 33.362
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_53.zip
Uploading the intermediate file: upload_file_53.zip
  20:25:01 STEP 1297 H=1297:00 +CPU= 18.068
  20:25:19 STEP 1298 H=1298:00 +CPU= 18.430
  20:25:48 STEP 1299 H=1299:00 +CPU= 28.554
  20:26:07 STEP 1300 H=1300:00 +CPU= 18.311
  20:26:25 STEP 1301 H=1301:00 +CPU= 17.786
  20:26:54 STEP 1302 H=1302:00 +CPU= 29.123
  20:27:12 STEP 1303 H=1303:00 +CPU= 17.885
  20:27:31 STEP 1304 H=1304:00 +CPU= 18.341
  20:28:00 STEP 1305 H=1305:00 +CPU= 28.607
  20:28:18 STEP 1306 H=1306:00 +CPU= 18.580
  20:28:36 STEP 1307 H=1307:00 +CPU= 17.955
  20:29:08 STEP 1308 H=1308:00 +CPU= 31.293
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001308
  20:29:26 STEP 1309 H=1309:00 +CPU= 17.953
  20:29:44 STEP 1310 H=1310:00 +CPU= 18.272
  20:30:13 STEP 1311 H=1311:00 +CPU= 28.326
  20:30:31 STEP 1312 H=1312:00 +CPU= 18.290
  20:30:49 STEP 1313 H=1313:00 +CPU= 17.788
  20:31:18 STEP 1314 H=1314:00 +CPU= 28.794
  20:31:35 STEP 1315 H=1315:00 +CPU= 17.755
  20:31:54 STEP 1316 H=1316:00 +CPU= 18.248
  20:32:22 STEP 1317 H=1317:00 +CPU= 28.196
  20:32:40 STEP 1318 H=1318:00 +CPU= 18.318
  20:32:59 STEP 1319 H=1319:00 +CPU= 18.203
  20:33:32 STEP 1320 H=1320:00 +CPU= 33.042
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_54.zip
Uploading the intermediate file: upload_file_54.zip
  20:33:50 STEP 1321 H=1321:00 +CPU= 17.836
  20:34:09 STEP 1322 H=1322:00 +CPU= 18.713
  20:34:37 STEP 1323 H=1323:00 +CPU= 28.376
  20:34:55 STEP 1324 H=1324:00 +CPU= 18.331
  20:35:14 STEP 1325 H=1325:00 +CPU= 18.020
  20:35:43 STEP 1326 H=1326:00 +CPU= 29.040
  20:36:01 STEP 1327 H=1327:00 +CPU= 17.822
  20:36:19 STEP 1328 H=1328:00 +CPU= 18.607
  20:36:47 STEP 1329 H=1329:00 +CPU= 28.206
  20:37:06 STEP 1330 H=1330:00 +CPU= 18.554
  20:37:24 STEP 1331 H=1331:00 +CPU= 17.809
  20:37:55 STEP 1332 H=1332:00 +CPU= 31.184
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001332
  20:38:13 STEP 1333 H=1333:00 +CPU= 17.668
  20:38:32 STEP 1334 H=1334:00 +CPU= 18.478
  20:39:00 STEP 1335 H=1335:00 +CPU= 28.347
  20:39:19 STEP 1336 H=1336:00 +CPU= 18.484
  20:39:36 STEP 1337 H=1337:00 +CPU= 17.697
  20:40:05 STEP 1338 H=1338:00 +CPU= 28.907
  20:40:23 STEP 1339 H=1339:00 +CPU= 17.660
  20:40:42 STEP 1340 H=1340:00 +CPU= 18.420
  20:41:10 STEP 1341 H=1341:00 +CPU= 28.162
  20:41:28 STEP 1342 H=1342:00 +CPU= 18.441
  20:41:47 STEP 1343 H=1343:00 +CPU= 18.160
  20:42:20 STEP 1344 H=1344:00 +CPU= 33.131
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_55.zip
Uploading the intermediate file: upload_file_55.zip
  20:42:38 STEP 1345 H=1345:00 +CPU= 17.737
  20:42:56 STEP 1346 H=1346:00 +CPU= 18.375
  20:43:24 STEP 1347 H=1347:00 +CPU= 28.293
  20:43:43 STEP 1348 H=1348:00 +CPU= 18.289
  20:44:01 STEP 1349 H=1349:00 +CPU= 17.663
  20:44:30 STEP 1350 H=1350:00 +CPU= 29.170
  20:44:48 STEP 1351 H=1351:00 +CPU= 17.914
  20:45:07 STEP 1352 H=1352:00 +CPU= 18.613
  20:45:36 STEP 1353 H=1353:00 +CPU= 28.476
  20:45:54 STEP 1354 H=1354:00 +CPU= 18.611
  20:46:13 STEP 1355 H=1355:00 +CPU= 18.019
  20:46:45 STEP 1356 H=1356:00 +CPU= 31.687
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001356
  20:47:03 STEP 1357 H=1357:00 +CPU= 17.887
  20:47:21 STEP 1358 H=1358:00 +CPU= 18.332
  20:47:50 STEP 1359 H=1359:00 +CPU= 28.398
  20:48:09 STEP 1360 H=1360:00 +CPU= 18.712
  20:48:27 STEP 1361 H=1361:00 +CPU= 17.919
  20:48:56 STEP 1362 H=1362:00 +CPU= 29.029
  20:49:14 STEP 1363 H=1363:00 +CPU= 17.918
  20:49:33 STEP 1364 H=1364:00 +CPU= 18.509
  20:50:02 STEP 1365 H=1365:00 +CPU= 28.520
  20:50:20 STEP 1366 H=1366:00 +CPU= 18.479
  20:50:39 STEP 1367 H=1367:00 +CPU= 18.259
  20:51:12 STEP 1368 H=1368:00 +CPU= 33.497
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_56.zip
Uploading the intermediate file: upload_file_56.zip
  20:51:31 STEP 1369 H=1369:00 +CPU= 18.009
  20:51:49 STEP 1370 H=1370:00 +CPU= 18.626
  20:52:18 STEP 1371 H=1371:00 +CPU= 28.722
  20:52:37 STEP 1372 H=1372:00 +CPU= 18.340
  20:52:55 STEP 1373 H=1373:00 +CPU= 17.980
  20:53:24 STEP 1374 H=1374:00 +CPU= 29.092
  20:53:42 STEP 1375 H=1375:00 +CPU= 17.995
  20:54:01 STEP 1376 H=1376:00 +CPU= 18.359
  20:54:30 STEP 1377 H=1377:00 +CPU= 28.596
  20:54:48 STEP 1378 H=1378:00 +CPU= 18.445
  20:55:07 STEP 1379 H=1379:00 +CPU= 17.987
  20:55:38 STEP 1380 H=1380:00 +CPU= 31.365
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001380
  20:55:56 STEP 1381 H=1381:00 +CPU= 18.035
  20:56:15 STEP 1382 H=1382:00 +CPU= 18.485
  20:56:44 STEP 1383 H=1383:00 +CPU= 28.366
  20:57:02 STEP 1384 H=1384:00 +CPU= 18.419
  20:57:20 STEP 1385 H=1385:00 +CPU= 17.750
  20:57:49 STEP 1386 H=1386:00 +CPU= 29.111
  20:58:07 STEP 1387 H=1387:00 +CPU= 18.005
  20:58:26 STEP 1388 H=1388:00 +CPU= 18.296
  20:58:55 STEP 1389 H=1389:00 +CPU= 28.456
  20:59:13 STEP 1390 H=1390:00 +CPU= 18.310
  20:59:32 STEP 1391 H=1391:00 +CPU= 18.424
  21:00:06 STEP 1392 H=1392:00 +CPU= 33.603
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_57.zip
Uploading the intermediate file: upload_file_57.zip
  21:00:24 STEP 1393 H=1393:00 +CPU= 18.060
  21:00:43 STEP 1394 H=1394:00 +CPU= 18.808
  21:01:12 STEP 1395 H=1395:00 +CPU= 28.796
  21:01:31 STEP 1396 H=1396:00 +CPU= 18.578
  21:01:49 STEP 1397 H=1397:00 +CPU= 17.920
  21:02:18 STEP 1398 H=1398:00 +CPU= 29.277
  21:02:37 STEP 1399 H=1399:00 +CPU= 17.926
  21:02:55 STEP 1400 H=1400:00 +CPU= 18.601
  21:03:24 STEP 1401 H=1401:00 +CPU= 28.528
  21:03:43 STEP 1402 H=1402:00 +CPU= 18.556
  21:04:01 STEP 1403 H=1403:00 +CPU= 18.008
  21:04:32 STEP 1404 H=1404:00 +CPU= 31.338
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001404
  21:04:50 STEP 1405 H=1405:00 +CPU= 17.813
  21:05:09 STEP 1406 H=1406:00 +CPU= 18.628
  21:05:38 STEP 1407 H=1407:00 +CPU= 28.392
  21:05:56 STEP 1408 H=1408:00 +CPU= 18.513
  21:06:15 STEP 1409 H=1409:00 +CPU= 18.079
  21:06:44 STEP 1410 H=1410:00 +CPU= 29.195
  21:07:02 STEP 1411 H=1411:00 +CPU= 17.815
  21:07:21 STEP 1412 H=1412:00 +CPU= 18.495
  21:07:50 STEP 1413 H=1413:00 +CPU= 28.741
  21:08:08 STEP 1414 H=1414:00 +CPU= 18.472
  21:08:27 STEP 1415 H=1415:00 +CPU= 18.322
  21:09:01 STEP 1416 H=1416:00 +CPU= 33.680
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_58.zip
Uploading the intermediate file: upload_file_58.zip
  21:09:19 STEP 1417 H=1417:00 +CPU= 17.953
  21:09:38 STEP 1418 H=1418:00 +CPU= 18.590
  21:10:07 STEP 1419 H=1419:00 +CPU= 28.557
  21:10:25 STEP 1420 H=1420:00 +CPU= 18.357
  21:10:43 STEP 1421 H=1421:00 +CPU= 17.947
  21:11:12 STEP 1422 H=1422:00 +CPU= 28.924
  21:11:30 STEP 1423 H=1423:00 +CPU= 17.825
  21:11:49 STEP 1424 H=1424:00 +CPU= 18.494
  21:12:17 STEP 1425 H=1425:00 +CPU= 28.460
  21:12:36 STEP 1426 H=1426:00 +CPU= 18.361
  21:12:54 STEP 1427 H=1427:00 +CPU= 17.826
  21:13:25 STEP 1428 H=1428:00 +CPU= 31.247
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001428
  21:13:43 STEP 1429 H=1429:00 +CPU= 17.844
  21:14:01 STEP 1430 H=1430:00 +CPU= 18.333
  21:14:30 STEP 1431 H=1431:00 +CPU= 28.355
  21:14:48 STEP 1432 H=1432:00 +CPU= 18.379
  21:15:06 STEP 1433 H=1433:00 +CPU= 17.793
  21:15:35 STEP 1434 H=1434:00 +CPU= 28.892
  21:15:53 STEP 1435 H=1435:00 +CPU= 17.819
  21:16:11 STEP 1436 H=1436:00 +CPU= 18.263
  21:16:39 STEP 1437 H=1437:00 +CPU= 28.331
  21:16:58 STEP 1438 H=1438:00 +CPU= 18.509
  21:17:16 STEP 1439 H=1439:00 +CPU= 18.250
  21:17:50 STEP 1440 H=1440:00 +CPU= 33.192
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_59.zip
Uploading the intermediate file: upload_file_59.zip
  21:18:08 STEP 1441 H=1441:00 +CPU= 17.808
  21:18:27 STEP 1442 H=1442:00 +CPU= 18.504
  21:18:55 STEP 1443 H=1443:00 +CPU= 28.548
  21:19:14 STEP 1444 H=1444:00 +CPU= 18.354
  21:19:32 STEP 1445 H=1445:00 +CPU= 17.844
  21:20:01 STEP 1446 H=1446:00 +CPU= 29.038
  21:20:19 STEP 1447 H=1447:00 +CPU= 17.863
  21:20:38 STEP 1448 H=1448:00 +CPU= 18.555
  21:21:07 STEP 1449 H=1449:00 +CPU= 28.879
  21:21:26 STEP 1450 H=1450:00 +CPU= 18.569
  21:21:44 STEP 1451 H=1451:00 +CPU= 18.131
  21:22:16 STEP 1452 H=1452:00 +CPU= 31.662
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001452
  21:22:34 STEP 1453 H=1453:00 +CPU= 17.977
  21:22:53 STEP 1454 H=1454:00 +CPU= 18.501
  21:23:22 STEP 1455 H=1455:00 +CPU= 28.513
  21:23:40 STEP 1456 H=1456:00 +CPU= 18.468
  21:23:59 STEP 1457 H=1457:00 +CPU= 18.102
  21:24:28 STEP 1458 H=1458:00 +CPU= 29.010
  21:24:46 STEP 1459 H=1459:00 +CPU= 17.942
  21:25:04 STEP 1460 H=1460:00 +CPU= 18.511
  21:25:33 STEP 1461 H=1461:00 +CPU= 28.396
  21:25:52 STEP 1462 H=1462:00 +CPU= 18.355
  21:26:10 STEP 1463 H=1463:00 +CPU= 18.505
  21:26:44 STEP 1464 H=1464:00 +CPU= 33.224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:27:02 STEP 1465 H=1465:00 +CPU= 18.048
Uploading trickle at timestep: 5270400
  21:27:21 STEP 1466 H=1466:00 +CPU= 18.776
  21:27:50 STEP 1467 H=1467:00 +CPU= 28.938
  21:28:09 STEP 1468 H=1468:00 +CPU= 18.520
  21:28:27 STEP 1469 H=1469:00 +CPU= 17.815
  21:28:57 STEP 1470 H=1470:00 +CPU= 29.228
  21:29:15 STEP 1471 H=1471:00 +CPU= 18.263
  21:29:34 STEP 1472 H=1472:00 +CPU= 18.498
  21:30:03 STEP 1473 H=1473:00 +CPU= 28.413
  21:30:21 STEP 1474 H=1474:00 +CPU= 18.620
  21:30:39 STEP 1475 H=1475:00 +CPU= 18.008
  21:31:11 STEP 1476 H=1476:00 +CPU= 31.304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001476
  21:31:29 STEP 1477 H=1477:00 +CPU= 17.991
  21:31:47 STEP 1478 H=1478:00 +CPU= 18.485
  21:32:16 STEP 1479 H=1479:00 +CPU= 28.359
  21:32:35 STEP 1480 H=1480:00 +CPU= 18.326
  21:32:53 STEP 1481 H=1481:00 +CPU= 17.990
  21:33:22 STEP 1482 H=1482:00 +CPU= 28.942
  21:33:40 STEP 1483 H=1483:00 +CPU= 17.851
  21:33:59 STEP 1484 H=1484:00 +CPU= 18.365
  21:34:27 STEP 1485 H=1485:00 +CPU= 28.446
  21:34:46 STEP 1486 H=1486:00 +CPU= 18.482
  21:35:05 STEP 1487 H=1487:00 +CPU= 18.456
  21:35:38 STEP 1488 H=1488:00 +CPU= 33.457
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_61.zip
Uploading the intermediate file: upload_file_61.zip
  21:35:56 STEP 1489 H=1489:00 +CPU= 17.915
  21:36:15 STEP 1490 H=1490:00 +CPU= 18.629
  21:36:44 STEP 1491 H=1491:00 +CPU= 28.497
  21:37:03 STEP 1492 H=1492:00 +CPU= 18.508
  21:37:21 STEP 1493 H=1493:00 +CPU= 17.830
  21:37:50 STEP 1494 H=1494:00 +CPU= 29.094
  21:38:08 STEP 1495 H=1495:00 +CPU= 18.000
  21:38:26 STEP 1496 H=1496:00 +CPU= 18.395
  21:38:55 STEP 1497 H=1497:00 +CPU= 28.435
  21:39:14 STEP 1498 H=1498:00 +CPU= 18.500
  21:39:32 STEP 1499 H=1499:00 +CPU= 17.864
  21:40:04 STEP 1500 H=1500:00 +CPU= 31.421
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001500
  21:40:22 STEP 1501 H=1501:00 +CPU= 17.960
  21:40:40 STEP 1502 H=1502:00 +CPU= 18.382
  21:41:09 STEP 1503 H=1503:00 +CPU= 28.462
  21:41:28 STEP 1504 H=1504:00 +CPU= 18.367
  21:41:46 STEP 1505 H=1505:00 +CPU= 17.785
  21:42:15 STEP 1506 H=1506:00 +CPU= 29.211
  21:42:33 STEP 1507 H=1507:00 +CPU= 18.007
  21:42:52 STEP 1508 H=1508:00 +CPU= 18.598
  21:43:21 STEP 1509 H=1509:00 +CPU= 28.397
  21:43:40 STEP 1510 H=1510:00 +CPU= 18.470
  21:43:58 STEP 1511 H=1511:00 +CPU= 18.378
  21:44:32 STEP 1512 H=1512:00 +CPU= 33.296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_62.zip
Uploading the intermediate file: upload_file_62.zip
  21:44:50 STEP 1513 H=1513:00 +CPU= 17.926
  21:45:08 STEP 1514 H=1514:00 +CPU= 18.384
  21:45:37 STEP 1515 H=1515:00 +CPU= 28.569
  21:45:55 STEP 1516 H=1516:00 +CPU= 18.393
  21:46:13 STEP 1517 H=1517:00 +CPU= 17.756
  21:46:42 STEP 1518 H=1518:00 +CPU= 29.083
  21:47:00 STEP 1519 H=1519:00 +CPU= 17.803
  21:47:18 STEP 1520 H=1520:00 +CPU= 18.307
  21:47:47 STEP 1521 H=1521:00 +CPU= 28.350
  21:48:05 STEP 1522 H=1522:00 +CPU= 18.323
  21:48:23 STEP 1523 H=1523:00 +CPU= 17.841
  21:48:54 STEP 1524 H=1524:00 +CPU= 31.283
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001524
  21:49:12 STEP 1525 H=1525:00 +CPU= 17.968
  21:49:31 STEP 1526 H=1526:00 +CPU= 18.275
  21:49:59 STEP 1527 H=1527:00 +CPU= 28.288
  21:50:17 STEP 1528 H=1528:00 +CPU= 18.251
  21:50:35 STEP 1529 H=1529:00 +CPU= 17.881
  21:51:04 STEP 1530 H=1530:00 +CPU= 28.843
  21:51:22 STEP 1531 H=1531:00 +CPU= 17.879
  21:51:41 STEP 1532 H=1532:00 +CPU= 18.316
  21:52:10 STEP 1533 H=1533:00 +CPU= 28.488
  21:52:28 STEP 1534 H=1534:00 +CPU= 18.336
  21:52:47 STEP 1535 H=1535:00 +CPU= 18.377
  21:53:20 STEP 1536 H=1536:00 +CPU= 33.357
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_63.zip
Uploading the intermediate file: upload_file_63.zip
  21:53:38 STEP 1537 H=1537:00 +CPU= 17.786
  21:53:57 STEP 1538 H=1538:00 +CPU= 18.444
  21:54:26 STEP 1539 H=1539:00 +CPU= 28.615
  21:54:44 STEP 1540 H=1540:00 +CPU= 18.305
  21:55:02 STEP 1541 H=1541:00 +CPU= 17.831
  21:55:32 STEP 1542 H=1542:00 +CPU= 29.194
  21:55:50 STEP 1543 H=1543:00 +CPU= 17.932
  21:56:09 STEP 1544 H=1544:00 +CPU= 18.394
  21:56:37 STEP 1545 H=1545:00 +CPU= 28.525
  21:56:56 STEP 1546 H=1546:00 +CPU= 18.586
  21:57:14 STEP 1547 H=1547:00 +CPU= 17.802
  21:57:46 STEP 1548 H=1548:00 +CPU= 31.629
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001548
  21:58:04 STEP 1549 H=1549:00 +CPU= 17.913
  21:58:22 STEP 1550 H=1550:00 +CPU= 18.335
  21:58:51 STEP 1551 H=1551:00 +CPU= 28.468
  21:59:09 STEP 1552 H=1552:00 +CPU= 18.316
  21:59:28 STEP 1553 H=1553:00 +CPU= 18.021
  21:59:57 STEP 1554 H=1554:00 +CPU= 28.953
  22:00:15 STEP 1555 H=1555:00 +CPU= 17.895
  22:00:34 STEP 1556 H=1556:00 +CPU= 18.528
  22:01:02 STEP 1557 H=1557:00 +CPU= 28.322
  22:01:21 STEP 1558 H=1558:00 +CPU= 18.487
  22:01:40 STEP 1559 H=1559:00 +CPU= 18.260
  22:02:14 STEP 1560 H=1560:00 +CPU= 33.522
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_64.zip
Uploading the intermediate file: upload_file_64.zip
  22:02:32 STEP 1561 H=1561:00 +CPU= 17.900
  22:02:50 STEP 1562 H=1562:00 +CPU= 18.537
  22:03:19 STEP 1563 H=1563:00 +CPU= 28.558
  22:03:38 STEP 1564 H=1564:00 +CPU= 18.434
  22:03:56 STEP 1565 H=1565:00 +CPU= 17.911
  22:04:25 STEP 1566 H=1566:00 +CPU= 29.085
  22:04:43 STEP 1567 H=1567:00 +CPU= 17.925
  22:05:02 STEP 1568 H=1568:00 +CPU= 18.313
  22:05:30 STEP 1569 H=1569:00 +CPU= 28.432
  22:05:49 STEP 1570 H=1570:00 +CPU= 18.384
  22:06:07 STEP 1571 H=1571:00 +CPU= 17.865
  22:06:39 STEP 1572 H=1572:00 +CPU= 31.399
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001572
  22:06:57 STEP 1573 H=1573:00 +CPU= 17.865
  22:07:15 STEP 1574 H=1574:00 +CPU= 18.360
  22:07:44 STEP 1575 H=1575:00 +CPU= 28.515
  22:08:03 STEP 1576 H=1576:00 +CPU= 18.400
  22:08:21 STEP 1577 H=1577:00 +CPU= 17.741
  22:08:50 STEP 1578 H=1578:00 +CPU= 29.118
  22:09:08 STEP 1579 H=1579:00 +CPU= 17.803
  22:09:26 STEP 1580 H=1580:00 +CPU= 18.294
  22:09:55 STEP 1581 H=1581:00 +CPU= 28.514
  22:10:14 STEP 1582 H=1582:00 +CPU= 18.532
  22:10:32 STEP 1583 H=1583:00 +CPU= 18.259
  22:11:05 STEP 1584 H=1584:00 +CPU= 33.089
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_65.zip
Uploading the intermediate file: upload_file_65.zip
  22:11:24 STEP 1585 H=1585:00 +CPU= 18.005
  22:11:42 STEP 1586 H=1586:00 +CPU= 18.518
  22:12:11 STEP 1587 H=1587:00 +CPU= 28.409
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.019] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221224:1674853944:13494.022] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221225:1674853945:13495.023] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2922:2922] [20230127:221225:1674853945:13495.023] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FB016D  vdfmain_                 1287  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0639
(argv4) batchid: 962
(argv5) wuid: 12179283
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d23869a3a55536f80480086ee0df6f9b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0639_1993050100_123_962_12179283.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0639_1993050100_123_962_12179283.zip
ic_ancil_file: ic_ancil_12179283
ifsdata_file: ifsdata_12179283
climate_data_file: clim_data_12179283
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ecc8382f78c890a2fe64ad6fd2e7de4a to: /var/lib/boinc-client/slots/3/ic_ancil_12179283.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12179283.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_df6bcccde39ca98de7f210213d60db73 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179283.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179283.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179283.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179283.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12179283.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12179283.xml
-- Model is restarting --
Adjusting last_iter, 1588, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12179283.xml
last_cpu_time: 37165
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675
The child process has been launched with process id: 2776
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:2776:2776] [20230128:120656:1674904016:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:07:39 STEP 1584 H=1584:00 +CPU= 31.680
  12:08:01 STEP 1585 H=1585:00 +CPU= 18.556
  12:08:20 STEP 1586 H=1586:00 +CPU= 18.815
  12:08:48 STEP 1587 H=1587:00 +CPU= 28.247
  12:09:06 STEP 1588 H=1588:00 +CPU= 18.317
  12:09:24 STEP 1589 H=1589:00 +CPU= 17.829
  12:09:53 STEP 1590 H=1590:00 +CPU= 28.911
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0639
(argv4) batchid: 962
(argv5) wuid: 12179283
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d23869a3a55536f80480086ee0df6f9b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0639_1993050100_123_962_12179283.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0639_1993050100_123_962_12179283.zip
ic_ancil_file: ic_ancil_12179283
ifsdata_file: ifsdata_12179283
climate_data_file: clim_data_12179283
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ecc8382f78c890a2fe64ad6fd2e7de4a to: /var/lib/boinc-client/slots/3/ic_ancil_12179283.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12179283.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_df6bcccde39ca98de7f210213d60db73 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179283.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12179283.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179283.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12179283.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12179283.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12179283.xml
-- Model is restarting --
Adjusting last_iter, 1591, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12179283.xml
last_cpu_time: 37334
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675
The child process has been launched with process id: 3958
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:obata-H87-HD3:1:1:3958:3958] [20230128:121104:1674904264:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:11:38 STEP 1584 H=1584:00 +CPU= 31.897
  12:11:56 STEP 1585 H=1585:00 +CPU= 17.661
  12:12:14 STEP 1586 H=1586:00 +CPU= 18.283
  12:12:43 STEP 1587 H=1587:00 +CPU= 28.373
  12:13:01 STEP 1588 H=1588:00 +CPU= 18.063
  12:13:19 STEP 1589 H=1589:00 +CPU= 17.758
  12:13:48 STEP 1590 H=1590:00 +CPU= 29.118
  12:14:06 STEP 1591 H=1591:00 +CPU= 17.646
  12:14:24 STEP 1592 H=1592:00 +CPU= 18.314
  12:14:53 STEP 1593 H=1593:00 +CPU= 28.520
  12:15:12 STEP 1594 H=1594:00 +CPU= 18.214
  12:15:30 STEP 1595 H=1595:00 +CPU= 17.829
  12:16:01 STEP 1596 H=1596:00 +CPU= 31.303
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001596
  12:16:19 STEP 1597 H=1597:00 +CPU= 17.631
  12:16:37 STEP 1598 H=1598:00 +CPU= 18.383
  12:17:07 STEP 1599 H=1599:00 +CPU= 29.085
  12:17:27 STEP 1600 H=1600:00 +CPU= 18.879
  12:17:45 STEP 1601 H=1601:00 +CPU= 18.232
  12:18:17 STEP 1602 H=1602:00 +CPU= 30.072
  12:18:37 STEP 1603 H=1603:00 +CPU= 18.431
  12:18:56 STEP 1604 H=1604:00 +CPU= 18.942
  12:19:27 STEP 1605 H=1605:00 +CPU= 29.623
  12:19:46 STEP 1606 H=1606:00 +CPU= 18.343
  12:20:05 STEP 1607 H=1607:00 +CPU= 18.555
  12:20:41 STEP 1608 H=1608:00 +CPU= 33.917
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179283/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0639_1993050100_123_962_12179283_1_r52646675_66.zip
Uploading the intermediate file: upload_file_66.zip
  12:21:00 STEP 1609 H=1609:00 +CPU= 17.894
  12:21:20 STEP 1610 H=1610:00 +CPU= 18.937
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:43:39 1509499 22303619 oifs_43r3_ps_0639_1993050100_123_962_12179283_1 5,270,400 34,458 0.0065
09 May 2023 16:43:33 1509499 22303619 oifs_43r3_ps_0639_1993050100_123_962_12179283_1 4,406,400 29,182 0.0066
09 May 2023 16:42:22 1509499 22303619 oifs_43r3_ps_0639_1993050100_123_962_12179283_1 3,456,000 22,845 0.0066
09 May 2023 16:41:12 1509499 22303619 oifs_43r3_ps_0639_1993050100_123_962_12179283_1 864,000 5,330 0.0062


©2024 climateprediction.net