climateprediction.net home page
Task 22256594

Task 22256594

Name oifs_43r3_ps_0513_1986050100_123_955_12172157_0
Workunit 12172157
Created 21 Dec 2022, 8:50:03 UTC
Sent 8 Jan 2023, 22:56:53 UTC
Report deadline 7 Feb 2023, 22:56:53 UTC
Received 10 Jan 2023, 0:04:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1501984
Run time 17 hours 33 min 19 sec
CPU time 16 hours 45 min 22 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 5.31 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,406.79 MB
Peak swap size 4,974.27 MB
Peak disk usage 2,164.71 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
  22:50:18 STEP 1247 H=1247:00 +CPU= 30.346
  22:51:04 STEP 1248 H=1248:00 +CPU= 45.244
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001248
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001248
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001248
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001224
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001224
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001224
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001236
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001236
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001236
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_51.zip
Uploading the intermediate file: upload_file_51.zip
  22:51:31 STEP 1249 H=1249:00 +CPU= 26.852
  22:51:59 STEP 1250 H=1250:00 +CPU= 27.719
  22:52:30 STEP 1251 H=1251:00 +CPU= 30.830
  22:53:01 STEP 1252 H=1252:00 +CPU= 29.747
  22:53:27 STEP 1253 H=1253:00 +CPU= 26.096
  22:54:12 STEP 1254 H=1254:00 +CPU= 42.761
  22:54:40 STEP 1255 H=1255:00 +CPU= 27.753
  22:55:06 STEP 1256 H=1256:00 +CPU= 26.400
  22:55:45 STEP 1257 H=1257:00 +CPU= 37.878
  22:56:13 STEP 1258 H=1258:00 +CPU= 28.478
  22:56:44 STEP 1259 H=1259:00 +CPU= 30.146
  22:57:26 STEP 1260 H=1260:00 +CPU= 41.268
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001260
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001260
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001260
  22:57:55 STEP 1261 H=1261:00 +CPU= 29.371
  22:58:20 STEP 1262 H=1262:00 +CPU= 24.881
  22:58:58 STEP 1263 H=1263:00 +CPU= 36.845
  22:59:29 STEP 1264 H=1264:00 +CPU= 30.755
  22:59:57 STEP 1265 H=1265:00 +CPU= 28.192
  23:00:39 STEP 1266 H=1266:00 +CPU= 41.187
  23:01:05 STEP 1267 H=1267:00 +CPU= 25.644
  23:01:34 STEP 1268 H=1268:00 +CPU= 29.065
  23:02:19 STEP 1269 H=1269:00 +CPU= 42.876
  23:02:49 STEP 1270 H=1270:00 +CPU= 29.445
  23:03:19 STEP 1271 H=1271:00 +CPU= 29.138
  23:04:10 STEP 1272 H=1272:00 +CPU= 49.252
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001272
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001272
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001272
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001248
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001248
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001248
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001260
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001260
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001260
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_52.zip
Uploading the intermediate file: upload_file_52.zip
  23:04:35 STEP 1273 H=1273:00 +CPU= 24.110
  23:05:02 STEP 1274 H=1274:00 +CPU= 26.922
  23:05:46 STEP 1275 H=1275:00 +CPU= 43.554
  23:06:16 STEP 1276 H=1276:00 +CPU= 29.158
  23:06:41 STEP 1277 H=1277:00 +CPU= 24.911
  23:07:22 STEP 1278 H=1278:00 +CPU= 40.344
  23:07:47 STEP 1279 H=1279:00 +CPU= 25.692
  23:08:12 STEP 1280 H=1280:00 +CPU= 24.713
  23:08:46 STEP 1281 H=1281:00 +CPU= 33.681
  23:09:13 STEP 1282 H=1282:00 +CPU= 26.518
  23:09:40 STEP 1283 H=1283:00 +CPU= 26.473
  23:10:30 STEP 1284 H=1284:00 +CPU= 49.154
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001284
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001284
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001284
  23:10:57 STEP 1285 H=1285:00 +CPU= 27.531
  23:11:27 STEP 1286 H=1286:00 +CPU= 29.294
  23:12:05 STEP 1287 H=1287:00 +CPU= 37.380
  23:12:34 STEP 1288 H=1288:00 +CPU= 29.193
  23:13:04 STEP 1289 H=1289:00 +CPU= 29.858
  23:13:51 STEP 1290 H=1290:00 +CPU= 45.749
  23:14:18 STEP 1291 H=1291:00 +CPU= 26.780
  23:14:47 STEP 1292 H=1292:00 +CPU= 29.449
  23:15:29 STEP 1293 H=1293:00 +CPU= 41.298
  23:15:59 STEP 1294 H=1294:00 +CPU= 29.447
  23:16:25 STEP 1295 H=1295:00 +CPU= 26.189
  23:17:11 STEP 1296 H=1296:00 +CPU= 45.306
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001296
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001296
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001296
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001272
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001272
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001272
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001284
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001284
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001284
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_53.zip
Uploading the intermediate file: upload_file_53.zip
  23:17:36 STEP 1297 H=1297:00 +CPU= 24.358
  23:18:00 STEP 1298 H=1298:00 +CPU= 24.578
  23:18:37 STEP 1299 H=1299:00 +CPU= 36.284
  23:19:06 STEP 1300 H=1300:00 +CPU= 28.663
  23:19:33 STEP 1301 H=1301:00 +CPU= 27.181
  23:20:20 STEP 1302 H=1302:00 +CPU= 45.957
  23:20:45 STEP 1303 H=1303:00 +CPU= 24.605
  23:21:12 STEP 1304 H=1304:00 +CPU= 26.794
  23:21:54 STEP 1305 H=1305:00 +CPU= 42.508
  23:22:22 STEP 1306 H=1306:00 +CPU= 27.338
  23:22:49 STEP 1307 H=1307:00 +CPU= 27.136
  23:23:37 STEP 1308 H=1308:00 +CPU= 47.385
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001308
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001308
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001308
  23:24:07 STEP 1309 H=1309:00 +CPU= 29.096
  23:24:37 STEP 1310 H=1310:00 +CPU= 29.838
  23:25:22 STEP 1311 H=1311:00 +CPU= 45.101
  23:25:50 STEP 1312 H=1312:00 +CPU= 27.349
  23:26:16 STEP 1313 H=1313:00 +CPU= 25.957
  23:27:00 STEP 1314 H=1314:00 +CPU= 43.250
  23:27:22 STEP 1315 H=1315:00 +CPU= 22.427
  23:27:49 STEP 1316 H=1316:00 +CPU= 26.827
  23:28:33 STEP 1317 H=1317:00 +CPU= 42.801
  23:29:03 STEP 1318 H=1318:00 +CPU= 30.656
  23:29:28 STEP 1319 H=1319:00 +CPU= 23.976
  23:30:14 STEP 1320 H=1320:00 +CPU= 45.515
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001320
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001320
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001320
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001296
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001296
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001296
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001308
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001308
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001308
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_54.zip
Uploading the intermediate file: upload_file_54.zip
  23:30:39 STEP 1321 H=1321:00 +CPU= 25.154
  23:31:08 STEP 1322 H=1322:00 +CPU= 28.621
  23:31:48 STEP 1323 H=1323:00 +CPU= 40.028
  23:32:18 STEP 1324 H=1324:00 +CPU= 29.233
  23:32:45 STEP 1325 H=1325:00 +CPU= 26.919
  23:33:32 STEP 1326 H=1326:00 +CPU= 46.300
  23:34:00 STEP 1327 H=1327:00 +CPU= 27.879
  23:34:30 STEP 1328 H=1328:00 +CPU= 29.683
  23:35:11 STEP 1329 H=1329:00 +CPU= 41.250
  23:35:40 STEP 1330 H=1330:00 +CPU= 28.232
  23:36:09 STEP 1331 H=1331:00 +CPU= 29.266
  23:36:53 STEP 1332 H=1332:00 +CPU= 43.626
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001332
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001332
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001332
  23:37:19 STEP 1333 H=1333:00 +CPU= 25.647
  23:37:47 STEP 1334 H=1334:00 +CPU= 27.503
  23:38:27 STEP 1335 H=1335:00 +CPU= 39.631
  23:38:55 STEP 1336 H=1336:00 +CPU= 27.643
  23:39:20 STEP 1337 H=1337:00 +CPU= 24.525
  23:40:03 STEP 1338 H=1338:00 +CPU= 42.847
  23:40:28 STEP 1339 H=1339:00 +CPU= 24.939
  23:40:55 STEP 1340 H=1340:00 +CPU= 27.147
  23:41:38 STEP 1341 H=1341:00 +CPU= 42.644
  23:42:07 STEP 1342 H=1342:00 +CPU= 28.753
  23:42:38 STEP 1343 H=1343:00 +CPU= 29.962
  23:43:27 STEP 1344 H=1344:00 +CPU= 48.964
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001344
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001344
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001344
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001320
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001320
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001320
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001332
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001332
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001332
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_55.zip
Uploading the intermediate file: upload_file_55.zip
  23:43:52 STEP 1345 H=1345:00 +CPU= 24.817
  23:44:23 STEP 1346 H=1346:00 +CPU= 30.502
  23:45:05 STEP 1347 H=1347:00 +CPU= 42.251
  23:45:33 STEP 1348 H=1348:00 +CPU= 27.331
  23:46:01 STEP 1349 H=1349:00 +CPU= 27.548
  23:46:41 STEP 1350 H=1350:00 +CPU= 40.291
  23:47:04 STEP 1351 H=1351:00 +CPU= 22.643
  23:47:31 STEP 1352 H=1352:00 +CPU= 26.503
  23:48:11 STEP 1353 H=1353:00 +CPU= 39.258
  23:48:40 STEP 1354 H=1354:00 +CPU= 29.229
  23:49:05 STEP 1355 H=1355:00 +CPU= 24.459
  23:49:45 STEP 1356 H=1356:00 +CPU= 39.585
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001356
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001356
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001356
  23:50:13 STEP 1357 H=1357:00 +CPU= 28.143
  23:50:39 STEP 1358 H=1358:00 +CPU= 25.255
  23:51:21 STEP 1359 H=1359:00 +CPU= 41.790
  23:51:48 STEP 1360 H=1360:00 +CPU= 27.296
  23:52:17 STEP 1361 H=1361:00 +CPU= 28.788
  23:53:03 STEP 1362 H=1362:00 +CPU= 45.089
  23:53:32 STEP 1363 H=1363:00 +CPU= 28.417
  23:54:00 STEP 1364 H=1364:00 +CPU= 28.573
  23:54:44 STEP 1365 H=1365:00 +CPU= 42.922
  23:55:13 STEP 1366 H=1366:00 +CPU= 29.351
  23:55:41 STEP 1367 H=1367:00 +CPU= 27.475
  23:56:25 STEP 1368 H=1368:00 +CPU= 43.484
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001368
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001368
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001368
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001344
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001344
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001344
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001356
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001356
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001356
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_56.zip
Uploading the intermediate file: upload_file_56.zip
  23:56:54 STEP 1369 H=1369:00 +CPU= 29.346
  23:57:22 STEP 1370 H=1370:00 +CPU= 27.561
  23:58:06 STEP 1371 H=1371:00 +CPU= 42.980
  23:58:31 STEP 1372 H=1372:00 +CPU= 25.364
  23:58:59 STEP 1373 H=1373:00 +CPU= 27.559
  23:59:38 STEP 1374 H=1374:00 +CPU= 38.623
  00:00:06 STEP 1375 H=1375:00 +CPU= 28.157
  00:00:34 STEP 1376 H=1376:00 +CPU= 27.527
  00:01:17 STEP 1377 H=1377:00 +CPU= 41.955
  00:01:46 STEP 1378 H=1378:00 +CPU= 28.873
  00:02:13 STEP 1379 H=1379:00 +CPU= 26.986
  00:03:01 STEP 1380 H=1380:00 +CPU= 47.635
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001380
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001380
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001380
  00:03:30 STEP 1381 H=1381:00 +CPU= 28.485
  00:03:59 STEP 1382 H=1382:00 +CPU= 29.215
  00:04:42 STEP 1383 H=1383:00 +CPU= 42.846
  00:05:12 STEP 1384 H=1384:00 +CPU= 29.328
  00:05:40 STEP 1385 H=1385:00 +CPU= 27.636
  00:06:18 STEP 1386 H=1386:00 +CPU= 38.140
  00:06:46 STEP 1387 H=1387:00 +CPU= 27.158
  00:07:17 STEP 1388 H=1388:00 +CPU= 30.480
  00:07:59 STEP 1389 H=1389:00 +CPU= 42.095
  00:08:30 STEP 1390 H=1390:00 +CPU= 30.261
  00:08:58 STEP 1391 H=1391:00 +CPU= 28.472
  00:09:49 STEP 1392 H=1392:00 +CPU= 49.906
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001392
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001392
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001392
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001368
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001368
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001368
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001380
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001380
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001380
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_57.zip
Uploading the intermediate file: upload_file_57.zip
  00:10:16 STEP 1393 H=1393:00 +CPU= 27.484
  00:10:43 STEP 1394 H=1394:00 +CPU= 25.841
  00:11:25 STEP 1395 H=1395:00 +CPU= 42.220
  00:11:52 STEP 1396 H=1396:00 +CPU= 26.161
  00:12:18 STEP 1397 H=1397:00 +CPU= 26.391
  00:13:04 STEP 1398 H=1398:00 +CPU= 45.075
  00:13:30 STEP 1399 H=1399:00 +CPU= 25.919
  00:13:58 STEP 1400 H=1400:00 +CPU= 27.904
  00:14:41 STEP 1401 H=1401:00 +CPU= 43.087
  00:15:10 STEP 1402 H=1402:00 +CPU= 28.086
  00:15:39 STEP 1403 H=1403:00 +CPU= 28.470
  00:16:23 STEP 1404 H=1404:00 +CPU= 43.648
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001404
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001404
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001404
  00:16:48 STEP 1405 H=1405:00 +CPU= 24.962
  00:17:14 STEP 1406 H=1406:00 +CPU= 26.204
  00:17:53 STEP 1407 H=1407:00 +CPU= 38.601
  00:18:24 STEP 1408 H=1408:00 +CPU= 30.234
  00:18:53 STEP 1409 H=1409:00 +CPU= 28.738
  00:19:35 STEP 1410 H=1410:00 +CPU= 42.184
  00:20:04 STEP 1411 H=1411:00 +CPU= 28.897
  00:20:33 STEP 1412 H=1412:00 +CPU= 28.156
  00:21:13 STEP 1413 H=1413:00 +CPU= 39.932
  00:21:39 STEP 1414 H=1414:00 +CPU= 25.807
  00:22:06 STEP 1415 H=1415:00 +CPU= 26.165
  00:22:57 STEP 1416 H=1416:00 +CPU= 50.927
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001416
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001416
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001416
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001392
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001392
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001392
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001404
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001404
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001404
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_58.zip
Uploading the intermediate file: upload_file_58.zip
  00:23:25 STEP 1417 H=1417:00 +CPU= 27.764
  00:23:53 STEP 1418 H=1418:00 +CPU= 27.539
  00:24:34 STEP 1419 H=1419:00 +CPU= 41.034
  00:24:58 STEP 1420 H=1420:00 +CPU= 24.058
  00:25:26 STEP 1421 H=1421:00 +CPU= 27.573
  00:26:11 STEP 1422 H=1422:00 +CPU= 43.910
  00:26:41 STEP 1423 H=1423:00 +CPU= 30.403
  00:27:13 STEP 1424 H=1424:00 +CPU= 31.796
  00:27:57 STEP 1425 H=1425:00 +CPU= 42.962
  00:28:26 STEP 1426 H=1426:00 +CPU= 29.056
  00:28:54 STEP 1427 H=1427:00 +CPU= 27.945
  00:29:42 STEP 1428 H=1428:00 +CPU= 47.848
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001428
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001428
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001428
  00:30:10 STEP 1429 H=1429:00 +CPU= 27.177
  00:30:34 STEP 1430 H=1430:00 +CPU= 24.040
  00:31:11 STEP 1431 H=1431:00 +CPU= 37.017
  00:31:41 STEP 1432 H=1432:00 +CPU= 28.815
  00:32:08 STEP 1433 H=1433:00 +CPU= 27.000
  00:32:51 STEP 1434 H=1434:00 +CPU= 42.854
  00:33:18 STEP 1435 H=1435:00 +CPU= 26.954
  00:33:44 STEP 1436 H=1436:00 +CPU= 25.682
  00:34:28 STEP 1437 H=1437:00 +CPU= 43.927
  00:34:59 STEP 1438 H=1438:00 +CPU= 30.401
  00:35:25 STEP 1439 H=1439:00 +CPU= 26.084
  00:36:14 STEP 1440 H=1440:00 +CPU= 47.986
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001440
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001440
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001440
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001416
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001416
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001416
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001428
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001428
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001428
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_59.zip
Uploading the intermediate file: upload_file_59.zip
  00:36:38 STEP 1441 H=1441:00 +CPU= 23.573
Uploading trickle at timestep: 5184000
  00:37:06 STEP 1442 H=1442:00 +CPU= 27.763
  00:37:51 STEP 1443 H=1443:00 +CPU= 44.393
  00:38:16 STEP 1444 H=1444:00 +CPU= 25.246
  00:38:46 STEP 1445 H=1445:00 +CPU= 29.523
  00:39:31 STEP 1446 H=1446:00 +CPU= 44.588
  00:39:59 STEP 1447 H=1447:00 +CPU= 27.668
  00:40:27 STEP 1448 H=1448:00 +CPU= 27.672
  00:41:12 STEP 1449 H=1449:00 +CPU= 44.570
  00:41:38 STEP 1450 H=1450:00 +CPU= 25.929
  00:42:07 STEP 1451 H=1451:00 +CPU= 28.539
  00:42:50 STEP 1452 H=1452:00 +CPU= 43.334
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001452
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001452
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001452
  00:43:26 STEP 1453 H=1453:00 +CPU= 31.045
  00:43:57 STEP 1454 H=1454:00 +CPU= 30.175
  00:44:36 STEP 1455 H=1455:00 +CPU= 39.275
  00:45:05 STEP 1456 H=1456:00 +CPU= 27.711
  00:45:36 STEP 1457 H=1457:00 +CPU= 28.866
  00:46:26 STEP 1458 H=1458:00 +CPU= 46.994
  00:46:53 STEP 1459 H=1459:00 +CPU= 26.903
  00:47:22 STEP 1460 H=1460:00 +CPU= 28.005
  00:48:03 STEP 1461 H=1461:00 +CPU= 40.532
  00:48:31 STEP 1462 H=1462:00 +CPU= 27.828
  00:49:02 STEP 1463 H=1463:00 +CPU= 30.866
  00:50:04 STEP 1464 H=1464:00 +CPU= 54.632
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001464
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001464
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001464
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001440
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001440
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001440
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001452
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001452
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001452
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_60.zip
Uploading the intermediate file: upload_file_60.zip
  00:50:34 STEP 1465 H=1465:00 +CPU= 28.853
  00:51:01 STEP 1466 H=1466:00 +CPU= 25.862
  00:51:36 STEP 1467 H=1467:00 +CPU= 34.569
  00:52:06 STEP 1468 H=1468:00 +CPU= 28.766
  00:52:34 STEP 1469 H=1469:00 +CPU= 26.955
  00:53:16 STEP 1470 H=1470:00 +CPU= 41.355
  00:53:44 STEP 1471 H=1471:00 +CPU= 28.077
  00:54:15 STEP 1472 H=1472:00 +CPU= 30.010
  00:54:56 STEP 1473 H=1473:00 +CPU= 39.999
  00:55:24 STEP 1474 H=1474:00 +CPU= 28.299
  00:55:52 STEP 1475 H=1475:00 +CPU= 27.418
  00:56:40 STEP 1476 H=1476:00 +CPU= 47.193
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001476
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001476
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001476
  00:57:06 STEP 1477 H=1477:00 +CPU= 24.942
  00:57:33 STEP 1478 H=1478:00 +CPU= 26.563
  00:58:16 STEP 1479 H=1479:00 +CPU= 42.310
  00:58:45 STEP 1480 H=1480:00 +CPU= 29.328
  00:59:13 STEP 1481 H=1481:00 +CPU= 27.476
  00:59:52 STEP 1482 H=1482:00 +CPU= 38.616
  01:00:22 STEP 1483 H=1483:00 +CPU= 29.245
  01:00:52 STEP 1484 H=1484:00 +CPU= 29.522
  01:01:31 STEP 1485 H=1485:00 +CPU= 39.318
  01:01:58 STEP 1486 H=1486:00 +CPU= 26.318
  01:02:23 STEP 1487 H=1487:00 +CPU= 24.215
  01:03:13 STEP 1488 H=1488:00 +CPU= 50.296
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001488
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001488
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001488
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001464
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001464
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001464
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001476
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001476
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001476
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_61.zip
Uploading the intermediate file: upload_file_61.zip
  01:03:44 STEP 1489 H=1489:00 +CPU= 30.016
  01:04:12 STEP 1490 H=1490:00 +CPU= 28.131
  01:04:56 STEP 1491 H=1491:00 +CPU= 43.524
  01:05:24 STEP 1492 H=1492:00 +CPU= 27.334
  01:05:50 STEP 1493 H=1493:00 +CPU= 26.506
  01:06:33 STEP 1494 H=1494:00 +CPU= 42.375
  01:06:59 STEP 1495 H=1495:00 +CPU= 25.144
  01:07:24 STEP 1496 H=1496:00 +CPU= 24.925
  01:08:06 STEP 1497 H=1497:00 +CPU= 41.590
  01:08:33 STEP 1498 H=1498:00 +CPU= 26.452
  01:09:02 STEP 1499 H=1499:00 +CPU= 29.357
  01:09:49 STEP 1500 H=1500:00 +CPU= 46.044
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001500
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001500
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001500
  01:10:16 STEP 1501 H=1501:00 +CPU= 26.644
  01:10:39 STEP 1502 H=1502:00 +CPU= 22.344
  01:11:20 STEP 1503 H=1503:00 +CPU= 40.328
  01:11:46 STEP 1504 H=1504:00 +CPU= 25.623
  01:12:12 STEP 1505 H=1505:00 +CPU= 26.322
  01:12:53 STEP 1506 H=1506:00 +CPU= 40.557
  01:13:14 STEP 1507 H=1507:00 +CPU= 20.297
  01:13:42 STEP 1508 H=1508:00 +CPU= 27.871
  01:14:22 STEP 1509 H=1509:00 +CPU= 39.069
  01:14:50 STEP 1510 H=1510:00 +CPU= 27.956
  01:15:14 STEP 1511 H=1511:00 +CPU= 22.967
  01:16:00 STEP 1512 H=1512:00 +CPU= 45.515
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001512
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001512
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001512
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001488
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001488
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001488
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001500
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001500
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001500
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_62.zip
Uploading the intermediate file: upload_file_62.zip
  01:16:26 STEP 1513 H=1513:00 +CPU= 25.742
  01:16:55 STEP 1514 H=1514:00 +CPU= 28.364
  01:17:36 STEP 1515 H=1515:00 +CPU= 40.484
  01:18:03 STEP 1516 H=1516:00 +CPU= 26.606
  01:18:33 STEP 1517 H=1517:00 +CPU= 28.787
  01:19:16 STEP 1518 H=1518:00 +CPU= 42.700
  01:19:43 STEP 1519 H=1519:00 +CPU= 26.204
  01:20:13 STEP 1520 H=1520:00 +CPU= 29.967
  01:20:55 STEP 1521 H=1521:00 +CPU= 40.102
  01:21:23 STEP 1522 H=1522:00 +CPU= 27.460
  01:21:58 STEP 1523 H=1523:00 +CPU= 32.217
  01:22:43 STEP 1524 H=1524:00 +CPU= 44.420
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001524
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001524
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001524
  01:23:10 STEP 1525 H=1525:00 +CPU= 26.633
  01:23:34 STEP 1526 H=1526:00 +CPU= 23.236
  01:24:16 STEP 1527 H=1527:00 +CPU= 40.831
  01:24:43 STEP 1528 H=1528:00 +CPU= 27.172
  01:25:09 STEP 1529 H=1529:00 +CPU= 24.932
  01:25:52 STEP 1530 H=1530:00 +CPU= 41.878
  01:26:16 STEP 1531 H=1531:00 +CPU= 24.181
  01:26:46 STEP 1532 H=1532:00 +CPU= 29.810
  01:27:27 STEP 1533 H=1533:00 +CPU= 40.504
  01:27:56 STEP 1534 H=1534:00 +CPU= 28.468
  01:28:23 STEP 1535 H=1535:00 +CPU= 26.942
  01:29:14 STEP 1536 H=1536:00 +CPU= 49.722
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001536
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001536
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001536
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001512
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001512
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001512
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001524
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001524
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001524
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_63.zip
Uploading the intermediate file: upload_file_63.zip
  01:29:41 STEP 1537 H=1537:00 +CPU= 26.470
  01:30:10 STEP 1538 H=1538:00 +CPU= 28.707
  01:30:52 STEP 1539 H=1539:00 +CPU= 41.343
  01:31:20 STEP 1540 H=1540:00 +CPU= 27.657
  01:31:49 STEP 1541 H=1541:00 +CPU= 28.654
  01:32:27 STEP 1542 H=1542:00 +CPU= 38.138
  01:32:55 STEP 1543 H=1543:00 +CPU= 26.904
  01:33:23 STEP 1544 H=1544:00 +CPU= 27.795
  01:34:09 STEP 1545 H=1545:00 +CPU= 44.163
  01:34:39 STEP 1546 H=1546:00 +CPU= 29.507
  01:35:05 STEP 1547 H=1547:00 +CPU= 25.317
  01:35:51 STEP 1548 H=1548:00 +CPU= 44.906
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001548
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001548
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001548
  01:36:17 STEP 1549 H=1549:00 +CPU= 25.560
  01:36:46 STEP 1550 H=1550:00 +CPU= 28.734
  01:37:28 STEP 1551 H=1551:00 +CPU= 41.121
  01:37:55 STEP 1552 H=1552:00 +CPU= 25.491
  01:38:20 STEP 1553 H=1553:00 +CPU= 24.855
  01:39:00 STEP 1554 H=1554:00 +CPU= 39.860
  01:39:22 STEP 1555 H=1555:00 +CPU= 21.179
  01:39:50 STEP 1556 H=1556:00 +CPU= 26.942
  01:40:33 STEP 1557 H=1557:00 +CPU= 42.107
  01:41:01 STEP 1558 H=1558:00 +CPU= 27.784
  01:41:28 STEP 1559 H=1559:00 +CPU= 27.197
  01:42:19 STEP 1560 H=1560:00 +CPU= 49.613
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001560
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001560
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001560
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001536
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001536
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001536
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMGGhq0f+001548
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMSHhq0f+001548
Adding to the zip: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157/ICMUAhq0f+001548
Zipping up the intermediate file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891_64.zip
Uploading the intermediate file: upload_file_64.zip
  01:42:48 STEP 1561 H=1561:00 +CPU= 28.539
  01:43:16 STEP 1562 H=1562:00 +CPU= 27.719
  01:43:58 STEP 1563 H=1563:00 +CPU= 41.981
  01:44:25 STEP 1564 H=1564:00 +CPU= 25.904
  01:44:48 STEP 1565 H=1565:00 +CPU= 23.057
  01:45:35 STEP 1566 H=1566:00 +CPU= 46.257
  01:46:01 STEP 1567 H=1567:00 +CPU= 26.238
  01:46:29 STEP 1568 H=1568:00 +CPU= 26.953
  01:47:10 STEP 1569 H=1569:00 +CPU= 40.341
  01:47:38 STEP 1570 H=1570:00 +CPU= 28.106
  01:48:06 STEP 1571 H=1571:00 +CPU= 27.240
  01:48:54 STEP 1572 H=1572:00 +CPU= 47.708
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMGGhq0f+001572
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMSHhq0f+001572
Moving to projects directory: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ICMUAhq0f+001572
  01:49:22 STEP 1573 H=1573:00 +CPU= 27.285
  01:49:50 STEP 1574 H=1574:00 +CPU= 27.668
  01:50:29 STEP 1575 H=1575:00 +CPU= 38.776
  01:50:56 STEP 1576 H=1576:00 +CPU= 25.915
  01:51:22 STEP 1577 H=1577:00 +CPU= 25.861
  01:52:05 STEP 1578 H=1578:00 +CPU= 42.579
  01:52:31 STEP 1579 H=1579:00 +CPU= 24.894
  01:53:00 STEP 1580 H=1580:00 +CPU= 28.924
  01:53:40 STEP 1581 H=1581:00 +CPU= 39.235
  01:54:07 STEP 1582 H=1582:00 +CPU= 26.082
  01:54:36 STEP 1583 H=1583:00 +CPU= 27.580
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015439:1673283279:62097.292] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2473MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62098.383] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62098.383] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62098.383] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62098.383] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62098.383] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2594 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62098.383] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015441:1673283281:62099.278] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.308] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.308] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.309] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.309] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.309] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        UPDTIM 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.309] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         UPDCLIE 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015442:1673283282:62099.309] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GRIB_API:IGRIB_NEW_FROM_FILE 
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015443:1673283283:62101.227] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:117335:117335] [20230110:015443:1673283283:62101.227] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC8C62  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DF7939  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DF02D3  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AF4254  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AF3FB2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AF5EF6  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AEFE86  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AE1B68  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AD6FE0  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000A05315  grib_api_interfac         601  grib_api_interface.F90
oifs_43r3_model.e  00000000014B011E  updclie_                  631  updclie.F90
oifs_43r3_model.e  000000000140A880  updtim_                   979  updtim.F90
oifs_43r3_model.e  000000000055C7B0  cnt4_                     592  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1986050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0513
(argv4) batchid: 955
(argv5) wuid: 12172157
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10
Project directory is: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172157
..mkdir for temp folder for results failed
Copying: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2e69e5ebacfca1d15a1737c21f6fdec4 to: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_ps_0513_1986050100_123_955_12172157.zip
Unzipping the namelist zip file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_ps_0513_1986050100_123_955_12172157.zip
ic_ancil_file: ic_ancil_12172157
ifsdata_file: ifsdata_12172157
climate_data_file: clim_data_12172157
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_ed8406aa2b79c29c53f4546374b4ac68 to: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ic_ancil_12172157.zip
Unzipping the IC ancils zip file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ic_ancil_12172157.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4d9de7b28c4ba6e10e4ee89a31d592bc to: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ifsdata/ifsdata_12172157.zip
Unzipping the ifsdata_zip file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/ifsdata/ifsdata_12172157.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/159l_2/clim_data_12172157.zip
Unzipping the climate data zip file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/159l_2/clim_data_12172157.zip
Checking for progress XML file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/progress_file_12172157.xml
Opened progress file ok : /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/progress_file_12172157.xml
-- Model is restarting --
Adjusting last_iter, 1583, to previous model restart step.
Creating progress file: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/progress_file_12172157.xml
last_cpu_time: 60286
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0513_1986050100_123_955_12172157_0_r357330891
The child process has been launched with process id: 16598
Executing the command: /mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080213:1673305333:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
 ABORT!    1 !! *** WAVE MODEL HAS ABORTED 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          INITMDL 
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.313] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           USERIN 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_model.exe' : sigcontextptr=0x7fff6fc8fb20
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/c3d79be7-9abe-4cfc-846c-23e6d095d443/boinc-client/slots/10/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.333] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.334] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.334] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.334] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x7c000040d6], nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080253:1673305373:40.334] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080254:1673305374:41.334] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:ubuntu-mss:1:1:16598:16598] [20230110:080254:1673305374:41.334] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC903B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008C2880  sdl_mod_mp_sdl_sr         100  sdl_mod.F90
oifs_43r3_model.e  000000000081D8AF  abor1_                     41  abor1.F90
oifs_43r3_model.e  00000000015F8ACA  mpabort_                   50  mpabort.F
oifs_43r3_model.e  00000000015D51C8  abort1_                    44  abort1.F
oifs_43r3_model.e  000000000166B07F  userin_                   553  userin.F
oifs_43r3_model.e  00000000015D7FF0  initmdl_                  452  initmdl.F
oifs_43r3_model.e  00000000008711E0  wavemdl_                  393  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:13:51 1501984 22256594 oifs_43r3_ps_0513_1986050100_123_955_12172157_0 5,184,000 55,690 0.0107
09 May 2023 16:13:45 1501984 22256594 oifs_43r3_ps_0513_1986050100_123_955_12172157_0 4,320,000 47,900 0.0111
09 May 2023 16:13:39 1501984 22256594 oifs_43r3_ps_0513_1986050100_123_955_12172157_0 3,456,000 40,072 0.0116
09 May 2023 16:13:33 1501984 22256594 oifs_43r3_ps_0513_1986050100_123_955_12172157_0 2,592,000 32,132 0.0124
09 May 2023 16:13:26 1501984 22256594 oifs_43r3_ps_0513_1986050100_123_955_12172157_0 1,728,000 21,715 0.0126
09 May 2023 16:13:19 1501984 22256594 oifs_43r3_ps_0513_1986050100_123_955_12172157_0 864,000 10,829 0.0125


©2024 climateprediction.net